Sep 07 12:57:40.637807 ip-10-0-142-216 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 07 12:57:40.637816 ip-10-0-142-216 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 07 12:57:40.637823 ip-10-0-142-216 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 07 12:57:40.638038 ip-10-0-142-216 systemd[1]: Failed to start Kubernetes Kubelet. Sep 07 12:57:49.852710 ip-10-0-142-216 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Sep 07 12:57:49.852725 ip-10-0-142-216 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f6afda53fb964a8f886774ea1b94d5f7 -- Sep 07 12:59:36.775060 ip-10-0-142-216 systemd[1]: Starting Kubernetes Kubelet... Sep 07 12:59:37.191709 ip-10-0-142-216 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 12:59:37.191709 ip-10-0-142-216 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 07 12:59:37.191709 ip-10-0-142-216 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 12:59:37.191709 ip-10-0-142-216 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 07 12:59:37.191709 ip-10-0-142-216 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 12:59:37.193732 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.193653 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 07 12:59:37.198763 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198748 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 12:59:37.198763 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198762 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198766 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198769 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198772 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198775 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198778 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198781 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198784 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198787 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198790 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198793 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198796 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198810 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198813 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198821 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198825 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198827 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198830 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198833 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198835 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 12:59:37.198837 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198838 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198841 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198844 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198847 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198850 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198853 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198855 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198858 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198860 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198863 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198866 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198869 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198871 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198875 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198879 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198883 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198885 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198889 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198892 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198894 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 12:59:37.199323 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198897 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198899 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198901 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198904 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198906 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198909 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198912 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198915 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198917 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198920 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198922 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198925 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198929 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198933 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198936 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198939 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198942 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198944 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198952 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 12:59:37.199825 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198955 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198958 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198961 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198964 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198967 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198969 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198972 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198975 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198978 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198981 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198984 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198986 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198989 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198991 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198994 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198996 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.198999 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199001 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199004 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199007 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 12:59:37.200302 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199010 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199013 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199015 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199018 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199020 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199023 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199025 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199030 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199034 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199037 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199040 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199043 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199045 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199048 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199051 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199054 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199056 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199059 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199061 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 12:59:37.200821 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199748 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199754 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199757 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199760 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199763 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199768 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199770 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199773 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199776 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199779 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199782 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199784 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199787 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199790 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199793 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199796 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199814 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199817 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199820 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199822 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 12:59:37.201362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199825 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199828 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199831 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199834 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199836 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199839 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199841 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199844 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199846 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199849 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199851 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199854 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199856 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199861 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199864 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199867 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199870 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199874 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199878 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 12:59:37.201893 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199881 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199884 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199887 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199889 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199892 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199896 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199898 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199901 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199904 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199907 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199910 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199912 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199915 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199918 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199921 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199923 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199926 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199928 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199931 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199933 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 12:59:37.202362 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199936 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199938 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199941 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199943 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199946 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199948 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199950 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199953 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199955 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199958 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199962 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199964 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199967 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199970 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199973 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199976 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199978 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199981 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199983 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 12:59:37.202867 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199986 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199989 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199991 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199994 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.199997 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200000 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200002 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200005 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200007 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200010 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200013 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200016 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200020 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200022 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200024 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200027 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200029 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200032 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200035 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200037 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 12:59:37.203335 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200040 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200410 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200415 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200418 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200421 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200424 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200427 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200439 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200442 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200445 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200448 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200450 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200453 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200456 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200458 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200461 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200463 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200466 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200468 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200471 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 12:59:37.203847 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200474 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200476 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200479 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200481 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200484 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200486 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200489 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200491 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200493 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200496 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200499 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200503 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200505 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200509 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200512 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200515 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200517 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200520 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200523 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200525 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 12:59:37.204341 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200528 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200530 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200532 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200535 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200537 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200540 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200542 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200545 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200548 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200550 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200553 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200556 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200558 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200561 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200563 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200566 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200568 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200571 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200575 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 12:59:37.204864 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200579 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200582 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200584 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200587 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200590 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200594 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200597 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200600 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200603 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200605 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200608 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200610 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200613 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200615 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200618 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200620 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200623 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200625 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200628 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200631 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 12:59:37.205338 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200633 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200636 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200639 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200641 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200644 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200648 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200651 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200654 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200657 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200660 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200662 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200665 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200669 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200671 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200674 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200676 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200680 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200683 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200687 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 12:59:37.205877 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200689 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.200692 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200765 2565 flags.go:64] FLAG: --address="0.0.0.0" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200823 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200831 2565 flags.go:64] FLAG: --anonymous-auth="true" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200836 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200841 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200844 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200849 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200854 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200858 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200861 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200864 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200868 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200871 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200874 2565 flags.go:64] FLAG: --cgroup-root="" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200877 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200880 2565 flags.go:64] FLAG: --client-ca-file="" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200883 2565 flags.go:64] FLAG: --cloud-provider="external" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200885 2565 flags.go:64] FLAG: --cluster-dns="[]" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200889 2565 flags.go:64] FLAG: --cluster-domain="" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200892 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200895 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200899 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 07 12:59:37.206381 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200903 2565 flags.go:64] FLAG: --container-log-max-files="5" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200906 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200909 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200912 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200916 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200920 2565 flags.go:64] FLAG: --contention-profiling="false" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200924 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200928 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200931 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200933 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200937 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200940 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200943 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200946 2565 flags.go:64] FLAG: --enable-load-reader="false" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200949 2565 flags.go:64] FLAG: --enable-server="true" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200952 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200957 2565 flags.go:64] FLAG: --event-burst="100" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200960 2565 flags.go:64] FLAG: --event-qps="50" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200963 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200966 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200969 2565 flags.go:64] FLAG: --eviction-hard="" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200973 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200976 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200979 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200982 2565 flags.go:64] FLAG: --eviction-soft="" Sep 07 12:59:37.206995 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200985 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200987 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200990 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200993 2565 flags.go:64] FLAG: --experimental-mounter-path="" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200996 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.200999 2565 flags.go:64] FLAG: --fail-swap-on="true" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201002 2565 flags.go:64] FLAG: --feature-gates="" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201006 2565 flags.go:64] FLAG: --file-check-frequency="20s" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201008 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201012 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201015 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201018 2565 flags.go:64] FLAG: --healthz-port="10248" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201022 2565 flags.go:64] FLAG: --help="false" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201025 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201029 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201032 2565 flags.go:64] FLAG: --http-check-frequency="20s" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201035 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201038 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201041 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201044 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201047 2565 flags.go:64] FLAG: --image-service-endpoint="" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201050 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201053 2565 flags.go:64] FLAG: --kube-api-burst="100" Sep 07 12:59:37.207587 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201056 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201059 2565 flags.go:64] FLAG: --kube-api-qps="50" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201062 2565 flags.go:64] FLAG: --kube-reserved="" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201065 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201068 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201071 2565 flags.go:64] FLAG: --kubelet-cgroups="" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201074 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201077 2565 flags.go:64] FLAG: --lock-file="" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201080 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201083 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201085 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201091 2565 flags.go:64] FLAG: --log-json-split-stream="false" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201093 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201096 2565 flags.go:64] FLAG: --log-text-split-stream="false" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201099 2565 flags.go:64] FLAG: --logging-format="text" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201102 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201105 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201108 2565 flags.go:64] FLAG: --manifest-url="" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201110 2565 flags.go:64] FLAG: --manifest-url-header="" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201114 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201117 2565 flags.go:64] FLAG: --max-open-files="1000000" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201124 2565 flags.go:64] FLAG: --max-pods="110" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201127 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201130 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201133 2565 flags.go:64] FLAG: --memory-manager-policy="None" Sep 07 12:59:37.208204 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201136 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201139 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201142 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201145 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201153 2565 flags.go:64] FLAG: --node-status-max-images="50" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201156 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201160 2565 flags.go:64] FLAG: --oom-score-adj="-999" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201163 2565 flags.go:64] FLAG: --pod-cidr="" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201166 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201171 2565 flags.go:64] FLAG: --pod-manifest-path="" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201174 2565 flags.go:64] FLAG: --pod-max-pids="-1" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201177 2565 flags.go:64] FLAG: --pods-per-core="0" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201180 2565 flags.go:64] FLAG: --port="10250" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201182 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201185 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b3e8e5403241800e" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201188 2565 flags.go:64] FLAG: --qos-reserved="" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201191 2565 flags.go:64] FLAG: --read-only-port="10255" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201194 2565 flags.go:64] FLAG: --register-node="true" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201197 2565 flags.go:64] FLAG: --register-with-taints="" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201200 2565 flags.go:64] FLAG: --registry-burst="10" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201203 2565 flags.go:64] FLAG: --registry-qps="5" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201206 2565 flags.go:64] FLAG: --reserved-cpus="" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201209 2565 flags.go:64] FLAG: --reserved-memory="" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201213 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201216 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 07 12:59:37.208846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201219 2565 flags.go:64] FLAG: --rotate-certificates="false" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201222 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201224 2565 flags.go:64] FLAG: --runonce="false" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201227 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201231 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201234 2565 flags.go:64] FLAG: --seccomp-default="false" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201237 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201240 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201243 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201246 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201249 2565 flags.go:64] FLAG: --storage-driver-password="root" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201251 2565 flags.go:64] FLAG: --storage-driver-secure="false" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201254 2565 flags.go:64] FLAG: --storage-driver-table="stats" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201257 2565 flags.go:64] FLAG: --storage-driver-user="root" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201260 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201263 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201267 2565 flags.go:64] FLAG: --system-cgroups="" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201270 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201275 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201278 2565 flags.go:64] FLAG: --tls-cert-file="" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201281 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201285 2565 flags.go:64] FLAG: --tls-min-version="" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201289 2565 flags.go:64] FLAG: --tls-private-key-file="" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201291 2565 flags.go:64] FLAG: --topology-manager-policy="none" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201294 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 07 12:59:37.209456 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201297 2565 flags.go:64] FLAG: --topology-manager-scope="container" Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201300 2565 flags.go:64] FLAG: --v="2" Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201304 2565 flags.go:64] FLAG: --version="false" Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201308 2565 flags.go:64] FLAG: --vmodule="" Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201311 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.201314 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201423 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201426 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201430 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201433 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201435 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201439 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201441 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201444 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201446 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201449 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201451 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201454 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201456 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201459 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201461 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201464 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 12:59:37.210115 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201466 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201469 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201472 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201475 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201477 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201480 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201483 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201487 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201490 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201492 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201495 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201498 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201501 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201503 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201506 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201508 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201511 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201515 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201517 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 12:59:37.210657 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201520 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201522 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201525 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201528 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201530 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201533 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201536 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201538 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201541 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201544 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201546 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201548 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201551 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201554 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201556 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201559 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201561 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201564 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201566 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201569 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 12:59:37.211134 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201574 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201576 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201579 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201581 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201584 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201587 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201589 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201592 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201594 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201597 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201600 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201602 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201605 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201609 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201612 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201615 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201617 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201620 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201622 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 12:59:37.211631 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201625 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201629 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201632 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201635 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201638 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201640 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201643 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201645 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201648 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201651 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201656 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201658 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201661 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201665 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201667 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201670 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201673 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201675 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201678 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 12:59:37.212236 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201681 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201683 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201686 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201688 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201692 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.201695 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.202430 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.210032 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.210048 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.210984 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.210990 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211047 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211052 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211055 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 12:59:37.212714 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211060 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211064 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211067 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211070 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211073 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211076 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211079 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211082 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211084 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211088 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211091 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211095 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211097 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211100 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211103 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211106 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211108 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211111 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211113 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 12:59:37.213087 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211116 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211118 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211121 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211123 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211126 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211129 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211131 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211135 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211138 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211141 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211144 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211147 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211149 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211152 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211154 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211157 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211159 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211162 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211164 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211166 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211169 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 12:59:37.213583 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211171 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211174 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211177 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211180 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211182 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211185 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211188 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211190 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211192 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211195 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211197 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211200 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211203 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211206 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211208 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211210 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211213 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211216 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211219 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 12:59:37.214108 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211221 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211224 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211227 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211229 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211232 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211234 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211237 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211241 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211245 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211248 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211251 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211254 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211258 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211262 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211264 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211267 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211269 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211272 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211274 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 12:59:37.215180 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211277 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211280 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211283 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211285 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211287 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211290 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211293 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211295 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211298 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211300 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211303 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211306 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211308 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211311 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211313 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211316 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211318 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 12:59:37.215650 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211321 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.211326 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211452 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211457 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211460 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211463 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211466 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211469 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211471 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211474 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211478 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211482 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211485 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211488 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211491 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 12:59:37.216139 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211493 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211496 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211498 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211501 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211504 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211506 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211509 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211511 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211514 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211517 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211519 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211522 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211524 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211527 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211529 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211531 2565 feature_gate.go:328] unrecognized feature gate: Example Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211534 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211537 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211539 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211541 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 12:59:37.216533 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211544 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211546 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211548 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211551 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211554 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211556 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211559 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211562 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211564 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211567 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211569 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211572 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211575 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211577 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211580 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211582 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211584 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211587 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211590 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211592 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 12:59:37.217043 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211595 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211597 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211600 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211602 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211605 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211607 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211611 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211614 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211617 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211620 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211623 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211625 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211628 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211631 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211633 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211636 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211639 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211643 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211647 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 12:59:37.217548 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211650 2565 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211652 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211655 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211657 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211660 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211663 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211665 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211668 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211670 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211672 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211675 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211677 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211680 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211683 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211685 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211687 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211690 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211692 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211695 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211697 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 12:59:37.218016 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211700 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211702 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211704 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211707 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211710 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211712 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:37.211714 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.211719 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.211828 2565 server.go:962] "Client rotation is on, will bootstrap in background" Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.214212 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.215306 2565 server.go:1019] "Starting client certificate rotation" Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.215409 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.215444 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 12:59:37.218496 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.218232 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 12:59:37.222846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.222824 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 12:59:37.235348 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.235330 2565 log.go:25] "Validated CRI v1 runtime API" Sep 07 12:59:37.238814 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.238789 2565 log.go:25] "Validated CRI v1 image API" Sep 07 12:59:37.239655 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.239639 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 07 12:59:37.243303 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.243285 2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a708efd6-9c34-422a-9b77-fbcb806f08f1:/dev/nvme0n1p4 dec22b91-14de-4075-8d36-ac82e571c87d:/dev/nvme0n1p3] Sep 07 12:59:37.243377 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.243301 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 07 12:59:37.246011 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.245991 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 12:59:37.248993 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.248871 2565 manager.go:216] Machine: {Timestamp:2026-09-07 12:59:37.247431505 +0000 UTC m=+0.367611061 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3054911 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec227fcb3f2941defd6e4796c92edcc9 SystemUUID:ec227fcb-3f29-41de-fd6e-4796c92edcc9 BootID:f6afda53-fb96-4a8f-8867-74ea1b94d5f7 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b5:32:b6:ab:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b5:32:b6:ab:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:35:04:b8:9b:b9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 07 12:59:37.248993 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.248988 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 07 12:59:37.249110 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.249095 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 07 12:59:37.252842 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.252815 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 07 12:59:37.252981 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.252844 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-216.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 07 12:59:37.253701 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.253691 2565 topology_manager.go:138] "Creating topology manager with none policy" Sep 07 12:59:37.253734 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.253703 2565 container_manager_linux.go:309] "Creating device plugin manager" Sep 07 12:59:37.253734 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.253711 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 12:59:37.254429 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.254418 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 12:59:37.256360 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.256350 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 07 12:59:37.260114 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.260103 2565 state_mem.go:36] "Initialized new in-memory state store" Sep 07 12:59:37.260228 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.260219 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 07 12:59:37.260274 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.260267 2565 kubelet.go:486] "Attempting to sync node with API server" Sep 07 12:59:37.260308 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.260277 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 07 12:59:37.260308 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.260287 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 07 12:59:37.260308 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.260303 2565 kubelet.go:398] "Adding apiserver pod source" Sep 07 12:59:37.260424 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.260316 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 07 12:59:37.260694 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.260680 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 12:59:37.260724 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.260703 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 12:59:37.263112 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.263096 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 07 12:59:37.269552 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.269535 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 07 12:59:37.269630 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.269574 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 07 12:59:37.269758 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.269746 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 07 12:59:37.269846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.269765 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 07 12:59:37.269846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.269775 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 07 12:59:37.269846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.269783 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 07 12:59:37.269846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.269794 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 07 12:59:37.269846 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.269818 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 07 12:59:37.270510 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.270492 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 07 12:59:37.270510 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.270511 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 07 12:59:37.270612 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.270522 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 07 12:59:37.270612 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.270531 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 07 12:59:37.270612 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.270543 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 07 12:59:37.270612 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.270554 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 07 12:59:37.272377 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.272365 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 07 12:59:37.272418 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.272378 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 07 12:59:37.274032 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.274013 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cgbv2" Sep 07 12:59:37.274714 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.274696 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-216.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 07 12:59:37.274749 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.274703 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 07 12:59:37.276348 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.276336 2565 server.go:1268] "Started kubelet" Sep 07 12:59:37.276448 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.276415 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 07 12:59:37.276498 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.276421 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 07 12:59:37.276498 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.276491 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 07 12:59:37.277121 ip-10-0-142-216 systemd[1]: Started Kubernetes Kubelet. Sep 07 12:59:37.278678 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.278662 2565 server.go:310] "Adding debug handlers to kubelet server" Sep 07 12:59:37.279075 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.279061 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 07 12:59:37.283344 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.283323 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 07 12:59:37.284421 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.284405 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-216.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 07 12:59:37.284922 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.283847 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-216.ec2.internal.18d30b5e72c11b5b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-216.ec2.internal,UID:ip-10-0-142-216.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-216.ec2.internal,},FirstTimestamp:2026-09-07 12:59:37.276308315 +0000 UTC m=+0.396487871,LastTimestamp:2026-09-07 12:59:37.276308315 +0000 UTC m=+0.396487871,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-216.ec2.internal,}" Sep 07 12:59:37.285123 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.285106 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cgbv2" Sep 07 12:59:37.285836 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.285824 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 07 12:59:37.286551 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.286532 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 07 12:59:37.287274 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.287253 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:37.287372 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287343 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 07 12:59:37.287372 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287364 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 07 12:59:37.287451 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287389 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 07 12:59:37.287506 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287495 2565 reconstruct.go:97] "Volume reconstruction finished" Sep 07 12:59:37.287506 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287506 2565 reconciler.go:29] "Reconciler: start to sync state" Sep 07 12:59:37.287569 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287535 2565 factory.go:55] Registering systemd factory Sep 07 12:59:37.287569 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287549 2565 factory.go:223] Registration of the systemd container factory successfully Sep 07 12:59:37.287740 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287727 2565 factory.go:153] Registering CRI-O factory Sep 07 12:59:37.287740 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287738 2565 factory.go:223] Registration of the crio container factory successfully Sep 07 12:59:37.287844 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287783 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 07 12:59:37.287844 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287796 2565 factory.go:103] Registering Raw factory Sep 07 12:59:37.287844 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.287829 2565 manager.go:1196] Started watching for new ooms in manager Sep 07 12:59:37.288105 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.288095 2565 manager.go:349] Starting recovery of all containers Sep 07 12:59:37.293467 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.293444 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 12:59:37.297010 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.296958 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 07 12:59:37.298697 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.298677 2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-216.ec2.internal\" not found" node="ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.300086 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.300070 2565 manager.go:354] Recovery completed Sep 07 12:59:37.304913 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.304901 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 12:59:37.307159 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.307145 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientMemory" Sep 07 12:59:37.307239 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.307171 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 12:59:37.307239 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.307181 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientPID" Sep 07 12:59:37.307678 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.307665 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 07 12:59:37.307678 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.307674 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 12:59:37.307795 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.307676 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 07 12:59:37.307795 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.307726 2565 state_mem.go:36] "Initialized new in-memory state store" Sep 07 12:59:37.308674 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.308657 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientMemory" Sep 07 12:59:37.308732 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.308687 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 12:59:37.308732 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.308701 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientPID" Sep 07 12:59:37.309086 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.309075 2565 policy_none.go:49] "None policy: Start" Sep 07 12:59:37.309126 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.309091 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 07 12:59:37.309126 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.309100 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 07 12:59:37.310078 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.310067 2565 policy_none.go:47] "Start" Sep 07 12:59:37.339409 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.339394 2565 manager.go:337] "Starting Device Plugin manager" Sep 07 12:59:37.355879 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.339431 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 07 12:59:37.355879 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.339444 2565 server.go:87] "Starting device plugin registration server" Sep 07 12:59:37.355879 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.339689 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 07 12:59:37.355879 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.339701 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 07 12:59:37.355879 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.339777 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 07 12:59:37.355879 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.339873 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 07 12:59:37.355879 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.339883 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 07 12:59:37.355879 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.340320 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 07 12:59:37.355879 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.340368 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:37.380171 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.380141 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 07 12:59:37.380292 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.380194 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 07 12:59:37.380292 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.380216 2565 kubelet.go:2448] "Starting kubelet main sync loop" Sep 07 12:59:37.380292 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.380249 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 07 12:59:37.383314 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.383297 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 12:59:37.440797 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.440773 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 12:59:37.441786 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.441702 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientMemory" Sep 07 12:59:37.441786 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.441734 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 12:59:37.441786 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.441747 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientPID" Sep 07 12:59:37.441786 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.441776 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.449346 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.449330 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.449412 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.449352 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-216.ec2.internal\": node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:37.480449 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.480419 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-216.ec2.internal"] Sep 07 12:59:37.480561 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.480516 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 12:59:37.481477 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.481462 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientMemory" Sep 07 12:59:37.481569 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.481496 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 12:59:37.481569 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.481511 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientPID" Sep 07 12:59:37.483707 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.483692 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 12:59:37.484279 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.484261 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.484358 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.484306 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 12:59:37.484426 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.484398 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientMemory" Sep 07 12:59:37.484426 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.484424 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 12:59:37.484506 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.484447 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientPID" Sep 07 12:59:37.485592 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.485579 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientMemory" Sep 07 12:59:37.485646 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.485605 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 12:59:37.485646 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.485615 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientPID" Sep 07 12:59:37.485750 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.485734 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:37.486524 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.486511 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.486603 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.486534 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 12:59:37.487187 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.487169 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientMemory" Sep 07 12:59:37.487276 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.487198 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 12:59:37.487276 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.487212 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeHasSufficientPID" Sep 07 12:59:37.512400 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.512386 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-216.ec2.internal\" not found" node="ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.516537 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.516523 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-216.ec2.internal\" not found" node="ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.585860 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.585834 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:37.588284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.588264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3fc91691ab22e8aac22ac5fac61eae7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal\" (UID: \"a3fc91691ab22e8aac22ac5fac61eae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.588379 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.588288 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e5b89f6dd594d68f12f82f4340ea7e5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-216.ec2.internal\" (UID: \"1e5b89f6dd594d68f12f82f4340ea7e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.588379 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.588307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3fc91691ab22e8aac22ac5fac61eae7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal\" (UID: \"a3fc91691ab22e8aac22ac5fac61eae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.686870 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.686845 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:37.689054 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.689041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3fc91691ab22e8aac22ac5fac61eae7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal\" (UID: \"a3fc91691ab22e8aac22ac5fac61eae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.689099 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.689073 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3fc91691ab22e8aac22ac5fac61eae7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal\" (UID: \"a3fc91691ab22e8aac22ac5fac61eae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.689099 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.689090 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e5b89f6dd594d68f12f82f4340ea7e5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-216.ec2.internal\" (UID: \"1e5b89f6dd594d68f12f82f4340ea7e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.689164 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.689114 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1e5b89f6dd594d68f12f82f4340ea7e5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-216.ec2.internal\" (UID: \"1e5b89f6dd594d68f12f82f4340ea7e5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.689164 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.689133 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3fc91691ab22e8aac22ac5fac61eae7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal\" (UID: \"a3fc91691ab22e8aac22ac5fac61eae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.689164 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.689142 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3fc91691ab22e8aac22ac5fac61eae7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal\" (UID: \"a3fc91691ab22e8aac22ac5fac61eae7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.787549 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.787467 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:37.814790 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.814758 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.819580 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:37.819561 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-216.ec2.internal" Sep 07 12:59:37.888280 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.888109 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:37.988545 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:37.988526 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:38.088998 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:38.088939 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:38.189361 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:38.189335 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:39.654434 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:39.654407 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 07 12:59:39.654764 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:39.654546 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 07 12:59:39.654764 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:39.654582 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 07 12:59:39.726955 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:39.726930 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 12:54:38 +0000 UTC" deadline="2028-03-17 04:50:46.85791484 +0000 UTC" Sep 07 12:59:39.726955 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:39.726955 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13359h51m7.130963191s" Sep 07 12:59:39.730172 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:39.730153 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:39.830628 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:39.830549 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:39.844899 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:39.844871 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 12:59:39.884950 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:39.884918 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fc91691ab22e8aac22ac5fac61eae7.slice/crio-6c05e30f0ebfbede676aa63467f00e1838093bdb44fb83db6b8a6bf31cccf517 WatchSource:0}: Error finding container 6c05e30f0ebfbede676aa63467f00e1838093bdb44fb83db6b8a6bf31cccf517: Status 404 returned error can't find the container with id 6c05e30f0ebfbede676aa63467f00e1838093bdb44fb83db6b8a6bf31cccf517 Sep 07 12:59:39.885850 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:39.885829 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5b89f6dd594d68f12f82f4340ea7e5.slice/crio-b8c9b6ce9c468b38e10db17751031fdc2bc8f1b60bef0826020c4a71c4206ddb WatchSource:0}: Error finding container b8c9b6ce9c468b38e10db17751031fdc2bc8f1b60bef0826020c4a71c4206ddb: Status 404 returned error can't find the container with id b8c9b6ce9c468b38e10db17751031fdc2bc8f1b60bef0826020c4a71c4206ddb Sep 07 12:59:39.890251 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:39.890238 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 12:59:39.931203 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:39.931180 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-216.ec2.internal\" not found" Sep 07 12:59:39.996426 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:39.996406 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 12:59:40.028036 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.028006 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" Sep 07 12:59:40.050512 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.050495 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 12:59:40.051426 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.051414 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-216.ec2.internal" Sep 07 12:59:40.072096 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.072074 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 12:59:40.587036 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.587004 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 12:59:40.691243 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.691194 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 12:59:40.701464 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.701437 2565 apiserver.go:52] "Watching apiserver" Sep 07 12:59:40.710331 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.710172 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 07 12:59:40.711229 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.711201 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-j7kfv","openshift-network-diagnostics/network-check-target-tq7v6","openshift-network-operator/iptables-alerter-4bh9v","openshift-ovn-kubernetes/ovnkube-node-qs8fs","kube-system/global-pull-secret-syncer-7nlgb","kube-system/kube-apiserver-proxy-ip-10-0-142-216.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp","openshift-dns/node-resolver-vswfs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal","openshift-multus/multus-additional-cni-plugins-fjbjz","openshift-multus/network-metrics-daemon-6wqht","kube-system/konnectivity-agent-zbw5l","openshift-cluster-node-tuning-operator/tuned-bwjlq","openshift-image-registry/node-ca-hjp2h"] Sep 07 12:59:40.714493 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.714068 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:40.714493 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:40.714144 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 12:59:40.716972 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.716221 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hjp2h" Sep 07 12:59:40.719937 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.719127 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4bh9v" Sep 07 12:59:40.719937 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.719469 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 07 12:59:40.721023 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.720120 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 07 12:59:40.721023 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.720445 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lbqzv\"" Sep 07 12:59:40.721654 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.721312 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 07 12:59:40.721654 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.721502 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 07 12:59:40.724067 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.723286 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6trkj\"" Sep 07 12:59:40.724067 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.723381 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 07 12:59:40.724067 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.723567 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.724285 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.724265 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 07 12:59:40.725719 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.725668 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.726310 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.726144 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 07 12:59:40.727434 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.727292 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 07 12:59:40.728052 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.728010 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 07 12:59:40.728233 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.728211 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zbw5l" Sep 07 12:59:40.730510 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.730491 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 07 12:59:40.731083 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.731063 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-td2kc\"" Sep 07 12:59:40.731732 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.731713 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s2pjd\"" Sep 07 12:59:40.732097 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.732068 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 07 12:59:40.732287 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.732254 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 07 12:59:40.732488 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.732469 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 07 12:59:40.732565 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.732490 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 07 12:59:40.732565 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.732490 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.732565 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.732553 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-df9ht\"" Sep 07 12:59:40.732738 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.732564 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 07 12:59:40.735820 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.735800 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vswfs" Sep 07 12:59:40.736659 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e73d458-8cb5-4913-9ad2-54f0615ef55d-serviceca\") pod \"node-ca-hjp2h\" (UID: \"9e73d458-8cb5-4913-9ad2-54f0615ef55d\") " pod="openshift-image-registry/node-ca-hjp2h" Sep 07 12:59:40.736659 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba93a5ab-9dad-48f1-8c35-d6ce0732ac78-host-slash\") pod \"iptables-alerter-4bh9v\" (UID: \"ba93a5ab-9dad-48f1-8c35-d6ce0732ac78\") " pod="openshift-network-operator/iptables-alerter-4bh9v" Sep 07 12:59:40.736659 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736538 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-cni-dir\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.736659 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736571 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-run-k8s-cni-cncf-io\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.736905 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736761 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 07 12:59:40.736905 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736805 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h7547\"" Sep 07 12:59:40.736905 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-hostroot\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.736905 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736871 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-modprobe-d\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.737100 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736965 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-systemd\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.737100 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736997 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-conf-dir\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.737100 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736999 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 07 12:59:40.737100 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737053 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-run-multus-certs\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.737100 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737086 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8mc\" (UniqueName: \"kubernetes.io/projected/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-kube-api-access-4g8mc\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.737332 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-sysctl-d\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.737332 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.736808 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 07 12:59:40.737332 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737164 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-run\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.737332 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737194 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-sys\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.737332 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737225 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96aef354-0582-4a9d-b427-4f347a152da2-agent-certs\") pod \"konnectivity-agent-zbw5l\" (UID: \"96aef354-0582-4a9d-b427-4f347a152da2\") " pod="kube-system/konnectivity-agent-zbw5l" Sep 07 12:59:40.737332 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737258 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96aef354-0582-4a9d-b427-4f347a152da2-konnectivity-ca\") pod \"konnectivity-agent-zbw5l\" (UID: \"96aef354-0582-4a9d-b427-4f347a152da2\") " pod="kube-system/konnectivity-agent-zbw5l" Sep 07 12:59:40.737332 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737285 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba93a5ab-9dad-48f1-8c35-d6ce0732ac78-iptables-alerter-script\") pod \"iptables-alerter-4bh9v\" (UID: \"ba93a5ab-9dad-48f1-8c35-d6ce0732ac78\") " pod="openshift-network-operator/iptables-alerter-4bh9v" Sep 07 12:59:40.737332 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-etc-kubernetes\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.737735 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737348 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-lib-modules\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.737735 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh2hw\" (UniqueName: \"kubernetes.io/projected/62d7435f-c609-4aea-b385-72e0d14076ba-kube-api-access-zh2hw\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:40.737735 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rsdk\" (UniqueName: \"kubernetes.io/projected/ba93a5ab-9dad-48f1-8c35-d6ce0732ac78-kube-api-access-2rsdk\") pod \"iptables-alerter-4bh9v\" (UID: \"ba93a5ab-9dad-48f1-8c35-d6ce0732ac78\") " pod="openshift-network-operator/iptables-alerter-4bh9v" Sep 07 12:59:40.737735 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737497 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-system-cni-dir\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.737735 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-var-lib-cni-bin\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.737735 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-sysconfig\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.737735 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737682 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-kubernetes\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.738042 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737739 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-host\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.738042 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737778 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0f073e8-242c-4bcf-83db-2564e842dd66-tmp\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.738042 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737801 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e73d458-8cb5-4913-9ad2-54f0615ef55d-host\") pod \"node-ca-hjp2h\" (UID: \"9e73d458-8cb5-4913-9ad2-54f0615ef55d\") " pod="openshift-image-registry/node-ca-hjp2h" Sep 07 12:59:40.738042 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737818 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqtdt\" (UniqueName: \"kubernetes.io/projected/9e73d458-8cb5-4913-9ad2-54f0615ef55d-kube-api-access-dqtdt\") pod \"node-ca-hjp2h\" (UID: \"9e73d458-8cb5-4913-9ad2-54f0615ef55d\") " pod="openshift-image-registry/node-ca-hjp2h" Sep 07 12:59:40.738042 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737836 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-socket-dir-parent\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.738042 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-sysctl-conf\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.738042 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-var-lib-kubelet\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.738042 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737960 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-tuned\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.738042 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.737988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-cnibin\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.738042 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.738012 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-cni-binary-copy\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.738405 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.738078 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-run-netns\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.738405 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.738103 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-var-lib-cni-multus\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.738405 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.738125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-var-lib-kubelet\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.738405 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.738143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-daemon-config\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.738405 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.738160 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:40.738405 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.738174 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkkw5\" (UniqueName: \"kubernetes.io/projected/d0f073e8-242c-4bcf-83db-2564e842dd66-kube-api-access-fkkw5\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.738832 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.738721 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.740790 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.740515 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kgmtr\"" Sep 07 12:59:40.740790 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.740518 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 07 12:59:40.740790 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.740569 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 07 12:59:40.742726 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.742705 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.744205 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.744184 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 07 12:59:40.745822 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.745523 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:40.745822 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:40.745639 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 12:59:40.745964 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.745845 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 07 12:59:40.746053 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.746038 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 07 12:59:40.746306 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.746288 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 07 12:59:40.746379 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.746333 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dthxs\"" Sep 07 12:59:40.746379 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.746292 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 07 12:59:40.747489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.747386 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 07 12:59:40.747489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.747465 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 07 12:59:40.748033 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.747901 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:40.748033 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:40.747964 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 12:59:40.748164 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.748148 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p9c22\"" Sep 07 12:59:40.748306 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.748276 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 12:59:40.748512 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.748477 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 07 12:59:40.824732 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.824665 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" event={"ID":"a3fc91691ab22e8aac22ac5fac61eae7","Type":"ContainerStarted","Data":"6c05e30f0ebfbede676aa63467f00e1838093bdb44fb83db6b8a6bf31cccf517"} Sep 07 12:59:40.825743 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.825722 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-216.ec2.internal" event={"ID":"1e5b89f6dd594d68f12f82f4340ea7e5","Type":"ContainerStarted","Data":"b8c9b6ce9c468b38e10db17751031fdc2bc8f1b60bef0826020c4a71c4206ddb"} Sep 07 12:59:40.828298 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.828282 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 07 12:59:40.838664 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e73d458-8cb5-4913-9ad2-54f0615ef55d-serviceca\") pod \"node-ca-hjp2h\" (UID: \"9e73d458-8cb5-4913-9ad2-54f0615ef55d\") " pod="openshift-image-registry/node-ca-hjp2h" Sep 07 12:59:40.838664 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-cni-dir\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.838802 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-run-k8s-cni-cncf-io\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.838802 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-hostroot\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.838802 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838761 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-hostroot\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.838915 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838829 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-run-k8s-cni-cncf-io\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.838915 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-systemd\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.838915 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838904 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-cni-dir\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.839053 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838917 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-conf-dir\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.839053 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8mc\" (UniqueName: \"kubernetes.io/projected/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-kube-api-access-4g8mc\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.839053 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838967 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-conf-dir\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.839053 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96aef354-0582-4a9d-b427-4f347a152da2-agent-certs\") pod \"konnectivity-agent-zbw5l\" (UID: \"96aef354-0582-4a9d-b427-4f347a152da2\") " pod="kube-system/konnectivity-agent-zbw5l" Sep 07 12:59:40.839053 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.838999 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-systemd\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.839053 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krgsj\" (UniqueName: \"kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj\") pod \"network-check-target-tq7v6\" (UID: \"783539e3-6dae-4567-ba41-f9627dddc085\") " pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:40.839053 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839042 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/338fa313-abb1-4bf6-8333-3e27d004296e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spn2b\" (UniqueName: \"kubernetes.io/projected/338fa313-abb1-4bf6-8333-3e27d004296e-kube-api-access-spn2b\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839082 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gwkzd" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2143bf9-8156-489e-9d51-cdee196d4ea7-kubelet-config\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-etc-kubernetes\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rsdk\" (UniqueName: \"kubernetes.io/projected/ba93a5ab-9dad-48f1-8c35-d6ce0732ac78-kube-api-access-2rsdk\") pod \"iptables-alerter-4bh9v\" (UID: \"ba93a5ab-9dad-48f1-8c35-d6ce0732ac78\") " pod="openshift-network-operator/iptables-alerter-4bh9v" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839206 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-etc-kubernetes\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839215 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-sys-fs\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5k72\" (UniqueName: \"kubernetes.io/projected/3af9c7e0-e966-4f88-b051-853c1e470a1b-kube-api-access-t5k72\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839275 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-slash\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839291 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-run-netns\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.839325 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839275 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 07 12:59:40.839949 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839690 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e73d458-8cb5-4913-9ad2-54f0615ef55d-serviceca\") pod \"node-ca-hjp2h\" (UID: \"9e73d458-8cb5-4913-9ad2-54f0615ef55d\") " pod="openshift-image-registry/node-ca-hjp2h" Sep 07 12:59:40.839949 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839793 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-cni-bin\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.839949 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-sysconfig\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.839949 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-host\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.839949 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839898 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-sysconfig\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.839949 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-socket-dir-parent\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.839949 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-sysctl-conf\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-var-lib-kubelet\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-tuned\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839983 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-host\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.839994 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-cnibin\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840019 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-run-netns\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840035 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-var-lib-kubelet\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-var-lib-cni-multus\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840000 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-socket-dir-parent\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840114 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-cnibin\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-var-lib-kubelet\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840162 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-sysctl-conf\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-daemon-config\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840200 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-kubelet\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840202 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-run-netns\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840239 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-run-openvswitch\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840247 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-var-lib-kubelet\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.840284 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840267 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkkw5\" (UniqueName: \"kubernetes.io/projected/d0f073e8-242c-4bcf-83db-2564e842dd66-kube-api-access-fkkw5\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840291 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7wsh\" (UniqueName: \"kubernetes.io/projected/2fddfa4d-1664-4139-8891-e7c6bc67fde4-kube-api-access-f7wsh\") pod \"node-resolver-vswfs\" (UID: \"2fddfa4d-1664-4139-8891-e7c6bc67fde4\") " pod="openshift-dns/node-resolver-vswfs" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-etc-selinux\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840342 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba93a5ab-9dad-48f1-8c35-d6ce0732ac78-host-slash\") pod \"iptables-alerter-4bh9v\" (UID: \"ba93a5ab-9dad-48f1-8c35-d6ce0732ac78\") " pod="openshift-network-operator/iptables-alerter-4bh9v" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-modprobe-d\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-sys\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840412 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96aef354-0582-4a9d-b427-4f347a152da2-konnectivity-ca\") pod \"konnectivity-agent-zbw5l\" (UID: \"96aef354-0582-4a9d-b427-4f347a152da2\") " pod="kube-system/konnectivity-agent-zbw5l" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840436 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-node-log\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2143bf9-8156-489e-9d51-cdee196d4ea7-dbus\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-run-multus-certs\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-sysctl-d\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-run\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840582 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-run\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840584 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-sys\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh2hw\" (UniqueName: \"kubernetes.io/projected/62d7435f-c609-4aea-b385-72e0d14076ba-kube-api-access-zh2hw\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840668 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-log-socket\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840693 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/338fa313-abb1-4bf6-8333-3e27d004296e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.841147 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840723 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-multus-daemon-config\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840747 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba93a5ab-9dad-48f1-8c35-d6ce0732ac78-iptables-alerter-script\") pod \"iptables-alerter-4bh9v\" (UID: \"ba93a5ab-9dad-48f1-8c35-d6ce0732ac78\") " pod="openshift-network-operator/iptables-alerter-4bh9v" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-lib-modules\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840795 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-run-ovn-kubernetes\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840817 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-modprobe-d\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840820 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-cni-netd\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840845 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-system-cni-dir\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba93a5ab-9dad-48f1-8c35-d6ce0732ac78-host-slash\") pod \"iptables-alerter-4bh9v\" (UID: \"ba93a5ab-9dad-48f1-8c35-d6ce0732ac78\") " pod="openshift-network-operator/iptables-alerter-4bh9v" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-var-lib-cni-bin\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fddfa4d-1664-4139-8891-e7c6bc67fde4-hosts-file\") pod \"node-resolver-vswfs\" (UID: \"2fddfa4d-1664-4139-8891-e7c6bc67fde4\") " pod="openshift-dns/node-resolver-vswfs" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-var-lib-cni-bin\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62cb5867-f697-4ee1-b2e5-d2c58e33a717-env-overrides\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.840946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/338fa313-abb1-4bf6-8333-3e27d004296e-system-cni-dir\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841153 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96aef354-0582-4a9d-b427-4f347a152da2-konnectivity-ca\") pod \"konnectivity-agent-zbw5l\" (UID: \"96aef354-0582-4a9d-b427-4f347a152da2\") " pod="kube-system/konnectivity-agent-zbw5l" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841221 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-lib-modules\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841227 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-run-multus-certs\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.841928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841305 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-system-cni-dir\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841336 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-kubernetes\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841350 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-sysctl-d\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841367 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0f073e8-242c-4bcf-83db-2564e842dd66-tmp\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841389 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-host-var-lib-cni-multus\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841410 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-socket-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841424 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-kubernetes\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fddfa4d-1664-4139-8891-e7c6bc67fde4-tmp-dir\") pod \"node-resolver-vswfs\" (UID: \"2fddfa4d-1664-4139-8891-e7c6bc67fde4\") " pod="openshift-dns/node-resolver-vswfs" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba93a5ab-9dad-48f1-8c35-d6ce0732ac78-iptables-alerter-script\") pod \"iptables-alerter-4bh9v\" (UID: \"ba93a5ab-9dad-48f1-8c35-d6ce0732ac78\") " pod="openshift-network-operator/iptables-alerter-4bh9v" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841475 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e73d458-8cb5-4913-9ad2-54f0615ef55d-host\") pod \"node-ca-hjp2h\" (UID: \"9e73d458-8cb5-4913-9ad2-54f0615ef55d\") " pod="openshift-image-registry/node-ca-hjp2h" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqtdt\" (UniqueName: \"kubernetes.io/projected/9e73d458-8cb5-4913-9ad2-54f0615ef55d-kube-api-access-dqtdt\") pod \"node-ca-hjp2h\" (UID: \"9e73d458-8cb5-4913-9ad2-54f0615ef55d\") " pod="openshift-image-registry/node-ca-hjp2h" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841506 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e73d458-8cb5-4913-9ad2-54f0615ef55d-host\") pod \"node-ca-hjp2h\" (UID: \"9e73d458-8cb5-4913-9ad2-54f0615ef55d\") " pod="openshift-image-registry/node-ca-hjp2h" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841554 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-systemd-units\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-etc-openvswitch\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841645 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-run-ovn\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62cb5867-f697-4ee1-b2e5-d2c58e33a717-ovn-node-metrics-cert\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.842489 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j28fm\" (UniqueName: \"kubernetes.io/projected/62cb5867-f697-4ee1-b2e5-d2c58e33a717-kube-api-access-j28fm\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.841749 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-cni-binary-copy\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.842204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.842245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-run-systemd\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.842278 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-var-lib-openvswitch\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.842312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62cb5867-f697-4ee1-b2e5-d2c58e33a717-ovnkube-script-lib\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.842339 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/338fa313-abb1-4bf6-8333-3e27d004296e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.842371 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/338fa313-abb1-4bf6-8333-3e27d004296e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.842402 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62cb5867-f697-4ee1-b2e5-d2c58e33a717-ovnkube-config\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.842432 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/338fa313-abb1-4bf6-8333-3e27d004296e-cnibin\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.842463 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-registration-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.842491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-device-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.843098 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-cni-binary-copy\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.843207 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:40.843136 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:40.843851 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:40.843244 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs podName:62d7435f-c609-4aea-b385-72e0d14076ba nodeName:}" failed. No retries permitted until 2026-09-07 12:59:41.343206901 +0000 UTC m=+3.023592666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs") pod "network-metrics-daemon-6wqht" (UID: "62d7435f-c609-4aea-b385-72e0d14076ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:40.844890 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.844085 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d0f073e8-242c-4bcf-83db-2564e842dd66-etc-tuned\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.846475 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.846454 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96aef354-0582-4a9d-b427-4f347a152da2-agent-certs\") pod \"konnectivity-agent-zbw5l\" (UID: \"96aef354-0582-4a9d-b427-4f347a152da2\") " pod="kube-system/konnectivity-agent-zbw5l" Sep 07 12:59:40.847054 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.847024 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0f073e8-242c-4bcf-83db-2564e842dd66-tmp\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.851942 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.851923 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gwkzd" Sep 07 12:59:40.858206 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.858187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8mc\" (UniqueName: \"kubernetes.io/projected/a761ff6c-7f18-4b78-ae99-6e8d13bf1995-kube-api-access-4g8mc\") pod \"multus-j7kfv\" (UID: \"a761ff6c-7f18-4b78-ae99-6e8d13bf1995\") " pod="openshift-multus/multus-j7kfv" Sep 07 12:59:40.860185 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.860164 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh2hw\" (UniqueName: \"kubernetes.io/projected/62d7435f-c609-4aea-b385-72e0d14076ba-kube-api-access-zh2hw\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:40.861230 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.861211 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rsdk\" (UniqueName: \"kubernetes.io/projected/ba93a5ab-9dad-48f1-8c35-d6ce0732ac78-kube-api-access-2rsdk\") pod \"iptables-alerter-4bh9v\" (UID: \"ba93a5ab-9dad-48f1-8c35-d6ce0732ac78\") " pod="openshift-network-operator/iptables-alerter-4bh9v" Sep 07 12:59:40.861936 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.861914 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkkw5\" (UniqueName: \"kubernetes.io/projected/d0f073e8-242c-4bcf-83db-2564e842dd66-kube-api-access-fkkw5\") pod \"tuned-bwjlq\" (UID: \"d0f073e8-242c-4bcf-83db-2564e842dd66\") " pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:40.862756 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.862737 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqtdt\" (UniqueName: \"kubernetes.io/projected/9e73d458-8cb5-4913-9ad2-54f0615ef55d-kube-api-access-dqtdt\") pod \"node-ca-hjp2h\" (UID: \"9e73d458-8cb5-4913-9ad2-54f0615ef55d\") " pod="openshift-image-registry/node-ca-hjp2h" Sep 07 12:59:40.942963 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.942930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgsj\" (UniqueName: \"kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj\") pod \"network-check-target-tq7v6\" (UID: \"783539e3-6dae-4567-ba41-f9627dddc085\") " pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:40.942963 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.942967 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/338fa313-abb1-4bf6-8333-3e27d004296e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.943199 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.942984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spn2b\" (UniqueName: \"kubernetes.io/projected/338fa313-abb1-4bf6-8333-3e27d004296e-kube-api-access-spn2b\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.943199 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2143bf9-8156-489e-9d51-cdee196d4ea7-kubelet-config\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:40.943199 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943089 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943199 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-sys-fs\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.943199 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5k72\" (UniqueName: \"kubernetes.io/projected/3af9c7e0-e966-4f88-b051-853c1e470a1b-kube-api-access-t5k72\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.943199 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943162 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-slash\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943199 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-run-netns\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943223 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-cni-bin\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943233 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2143bf9-8156-489e-9d51-cdee196d4ea7-kubelet-config\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-kubelet\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943272 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-run-openvswitch\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7wsh\" (UniqueName: \"kubernetes.io/projected/2fddfa4d-1664-4139-8891-e7c6bc67fde4-kube-api-access-f7wsh\") pod \"node-resolver-vswfs\" (UID: \"2fddfa4d-1664-4139-8891-e7c6bc67fde4\") " pod="openshift-dns/node-resolver-vswfs" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943295 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-run-netns\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-etc-selinux\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-node-log\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943367 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2143bf9-8156-489e-9d51-cdee196d4ea7-dbus\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943394 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-etc-selinux\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943402 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-log-socket\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-kubelet\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-slash\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/338fa313-abb1-4bf6-8333-3e27d004296e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943497 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:40.943545 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943425 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-log-socket\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2143bf9-8156-489e-9d51-cdee196d4ea7-dbus\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-run-ovn-kubernetes\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943557 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-run-openvswitch\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943557 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-sys-fs\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:40.943609 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-cni-netd\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-run-ovn-kubernetes\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:40.943674 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret podName:b2143bf9-8156-489e-9d51-cdee196d4ea7 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:41.443655741 +0000 UTC m=+3.124041510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret") pod "global-pull-secret-syncer-7nlgb" (UID: "b2143bf9-8156-489e-9d51-cdee196d4ea7") : object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fddfa4d-1664-4139-8891-e7c6bc67fde4-hosts-file\") pod \"node-resolver-vswfs\" (UID: \"2fddfa4d-1664-4139-8891-e7c6bc67fde4\") " pod="openshift-dns/node-resolver-vswfs" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943700 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-cni-netd\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62cb5867-f697-4ee1-b2e5-d2c58e33a717-env-overrides\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/338fa313-abb1-4bf6-8333-3e27d004296e-system-cni-dir\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-socket-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943793 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fddfa4d-1664-4139-8891-e7c6bc67fde4-tmp-dir\") pod \"node-resolver-vswfs\" (UID: \"2fddfa4d-1664-4139-8891-e7c6bc67fde4\") " pod="openshift-dns/node-resolver-vswfs" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-node-log\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943846 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-systemd-units\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-etc-openvswitch\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943893 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/338fa313-abb1-4bf6-8333-3e27d004296e-system-cni-dir\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-run-ovn\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943943 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2fddfa4d-1664-4139-8891-e7c6bc67fde4-hosts-file\") pod \"node-resolver-vswfs\" (UID: \"2fddfa4d-1664-4139-8891-e7c6bc67fde4\") " pod="openshift-dns/node-resolver-vswfs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62cb5867-f697-4ee1-b2e5-d2c58e33a717-ovn-node-metrics-cert\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.943982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j28fm\" (UniqueName: \"kubernetes.io/projected/62cb5867-f697-4ee1-b2e5-d2c58e33a717-kube-api-access-j28fm\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944004 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/338fa313-abb1-4bf6-8333-3e27d004296e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-run-systemd\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-run-systemd\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944055 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-var-lib-openvswitch\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-host-cni-bin\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62cb5867-f697-4ee1-b2e5-d2c58e33a717-ovnkube-script-lib\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-var-lib-openvswitch\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944122 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/338fa313-abb1-4bf6-8333-3e27d004296e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/338fa313-abb1-4bf6-8333-3e27d004296e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.944982 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62cb5867-f697-4ee1-b2e5-d2c58e33a717-ovnkube-config\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944202 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/338fa313-abb1-4bf6-8333-3e27d004296e-cnibin\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-registration-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944254 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-systemd-units\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-device-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944302 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-etc-openvswitch\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944321 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-socket-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944329 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-device-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944202 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fddfa4d-1664-4139-8891-e7c6bc67fde4-tmp-dir\") pod \"node-resolver-vswfs\" (UID: \"2fddfa4d-1664-4139-8891-e7c6bc67fde4\") " pod="openshift-dns/node-resolver-vswfs" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944361 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62cb5867-f697-4ee1-b2e5-d2c58e33a717-run-ovn\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944369 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3af9c7e0-e966-4f88-b051-853c1e470a1b-registration-dir\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944364 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/338fa313-abb1-4bf6-8333-3e27d004296e-cnibin\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/338fa313-abb1-4bf6-8333-3e27d004296e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62cb5867-f697-4ee1-b2e5-d2c58e33a717-env-overrides\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/338fa313-abb1-4bf6-8333-3e27d004296e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/338fa313-abb1-4bf6-8333-3e27d004296e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944811 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62cb5867-f697-4ee1-b2e5-d2c58e33a717-ovnkube-config\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.945779 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.944816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62cb5867-f697-4ee1-b2e5-d2c58e33a717-ovnkube-script-lib\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.946552 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.946531 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62cb5867-f697-4ee1-b2e5-d2c58e33a717-ovn-node-metrics-cert\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.955781 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:40.955762 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 12:59:40.955781 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:40.955783 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 12:59:40.955940 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:40.955792 2565 projected.go:196] Error preparing data for projected volume kube-api-access-krgsj for pod openshift-network-diagnostics/network-check-target-tq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:40.955940 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:40.955846 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj podName:783539e3-6dae-4567-ba41-f9627dddc085 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:41.455834727 +0000 UTC m=+3.136220474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-krgsj" (UniqueName: "kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj") pod "network-check-target-tq7v6" (UID: "783539e3-6dae-4567-ba41-f9627dddc085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:40.958302 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.958278 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spn2b\" (UniqueName: \"kubernetes.io/projected/338fa313-abb1-4bf6-8333-3e27d004296e-kube-api-access-spn2b\") pod \"multus-additional-cni-plugins-fjbjz\" (UID: \"338fa313-abb1-4bf6-8333-3e27d004296e\") " pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:40.960397 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.960370 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j28fm\" (UniqueName: \"kubernetes.io/projected/62cb5867-f697-4ee1-b2e5-d2c58e33a717-kube-api-access-j28fm\") pod \"ovnkube-node-qs8fs\" (UID: \"62cb5867-f697-4ee1-b2e5-d2c58e33a717\") " pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:40.960496 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.960430 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5k72\" (UniqueName: \"kubernetes.io/projected/3af9c7e0-e966-4f88-b051-853c1e470a1b-kube-api-access-t5k72\") pod \"aws-ebs-csi-driver-node-tclkp\" (UID: \"3af9c7e0-e966-4f88-b051-853c1e470a1b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:40.960658 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:40.960638 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7wsh\" (UniqueName: \"kubernetes.io/projected/2fddfa4d-1664-4139-8891-e7c6bc67fde4-kube-api-access-f7wsh\") pod \"node-resolver-vswfs\" (UID: \"2fddfa4d-1664-4139-8891-e7c6bc67fde4\") " pod="openshift-dns/node-resolver-vswfs" Sep 07 12:59:41.031579 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.031544 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hjp2h" Sep 07 12:59:41.043921 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.043899 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4bh9v" Sep 07 12:59:41.054537 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.054520 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" Sep 07 12:59:41.061352 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.061333 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j7kfv" Sep 07 12:59:41.069005 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.068986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" Sep 07 12:59:41.077632 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.077604 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zbw5l" Sep 07 12:59:41.085166 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.085142 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vswfs" Sep 07 12:59:41.092928 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.092882 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 12:59:41.098526 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.098507 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fjbjz" Sep 07 12:59:41.346464 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.346386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:41.346633 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:41.346527 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:41.346633 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:41.346605 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs podName:62d7435f-c609-4aea-b385-72e0d14076ba nodeName:}" failed. No retries permitted until 2026-09-07 12:59:42.346577751 +0000 UTC m=+4.026963502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs") pod "network-metrics-daemon-6wqht" (UID: "62d7435f-c609-4aea-b385-72e0d14076ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:41.446918 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.446888 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:41.447061 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:41.447031 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:41.447110 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:41.447099 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret podName:b2143bf9-8156-489e-9d51-cdee196d4ea7 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:42.447082454 +0000 UTC m=+4.127468223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret") pod "global-pull-secret-syncer-7nlgb" (UID: "b2143bf9-8156-489e-9d51-cdee196d4ea7") : object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:41.548190 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.548154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgsj\" (UniqueName: \"kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj\") pod \"network-check-target-tq7v6\" (UID: \"783539e3-6dae-4567-ba41-f9627dddc085\") " pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:41.548371 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:41.548351 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 12:59:41.548419 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:41.548377 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 12:59:41.548419 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:41.548387 2565 projected.go:196] Error preparing data for projected volume kube-api-access-krgsj for pod openshift-network-diagnostics/network-check-target-tq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:41.548482 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:41.548444 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj podName:783539e3-6dae-4567-ba41-f9627dddc085 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:42.548429735 +0000 UTC m=+4.228815507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-krgsj" (UniqueName: "kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj") pod "network-check-target-tq7v6" (UID: "783539e3-6dae-4567-ba41-f9627dddc085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:41.820401 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.820376 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:41.820838 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:41.820480 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 12:59:41.826866 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:41.826829 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338fa313_abb1_4bf6_8333_3e27d004296e.slice/crio-6ef573fb37de4a0ee060cb8296595c9aa7347913476462a82b0e5b7a5eded642 WatchSource:0}: Error finding container 6ef573fb37de4a0ee060cb8296595c9aa7347913476462a82b0e5b7a5eded642: Status 404 returned error can't find the container with id 6ef573fb37de4a0ee060cb8296595c9aa7347913476462a82b0e5b7a5eded642 Sep 07 12:59:41.853167 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:41.853140 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cb5867_f697_4ee1_b2e5_d2c58e33a717.slice/crio-4c4dcbb44cd40247a72477ff1e3d0cdf69c7e88038aa90203aee4ffd587aea4b WatchSource:0}: Error finding container 4c4dcbb44cd40247a72477ff1e3d0cdf69c7e88038aa90203aee4ffd587aea4b: Status 404 returned error can't find the container with id 4c4dcbb44cd40247a72477ff1e3d0cdf69c7e88038aa90203aee4ffd587aea4b Sep 07 12:59:41.853824 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:41.853796 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e73d458_8cb5_4913_9ad2_54f0615ef55d.slice/crio-8fcf3a19354f4ed5f3bccf5ce413e977756837e66a41febdefbf7e8d82394da2 WatchSource:0}: Error finding container 8fcf3a19354f4ed5f3bccf5ce413e977756837e66a41febdefbf7e8d82394da2: Status 404 returned error can't find the container with id 8fcf3a19354f4ed5f3bccf5ce413e977756837e66a41febdefbf7e8d82394da2 Sep 07 12:59:41.854487 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:41.854450 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fddfa4d_1664_4139_8891_e7c6bc67fde4.slice/crio-360180427b0b9a76e1058fd549bfbc9bbafc093d9c4056be5066765383709454 WatchSource:0}: Error finding container 360180427b0b9a76e1058fd549bfbc9bbafc093d9c4056be5066765383709454: Status 404 returned error can't find the container with id 360180427b0b9a76e1058fd549bfbc9bbafc093d9c4056be5066765383709454 Sep 07 12:59:41.855788 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:41.855671 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba93a5ab_9dad_48f1_8c35_d6ce0732ac78.slice/crio-789236c1632a75fa46806781d301b9517ca116b12915686fe70571b520555de7 WatchSource:0}: Error finding container 789236c1632a75fa46806781d301b9517ca116b12915686fe70571b520555de7: Status 404 returned error can't find the container with id 789236c1632a75fa46806781d301b9517ca116b12915686fe70571b520555de7 Sep 07 12:59:41.856001 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.855961 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 12:54:40 +0000 UTC" deadline="2028-05-06 23:23:07.430246147 +0000 UTC" Sep 07 12:59:41.856001 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:41.855992 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14578h23m25.574258952s" Sep 07 12:59:41.857018 ip-10-0-142-216 kubenswrapper[2565]: W0907 12:59:41.856975 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f073e8_242c_4bcf_83db_2564e842dd66.slice/crio-0e20d8c921ce565cf86ca165f80e94827e7056d51206f01272862b3b60c70f8a WatchSource:0}: Error finding container 0e20d8c921ce565cf86ca165f80e94827e7056d51206f01272862b3b60c70f8a: Status 404 returned error can't find the container with id 0e20d8c921ce565cf86ca165f80e94827e7056d51206f01272862b3b60c70f8a Sep 07 12:59:42.355630 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.355343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:42.355788 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:42.355719 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:42.355788 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:42.355784 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs podName:62d7435f-c609-4aea-b385-72e0d14076ba nodeName:}" failed. No retries permitted until 2026-09-07 12:59:44.355765184 +0000 UTC m=+6.036150938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs") pod "network-metrics-daemon-6wqht" (UID: "62d7435f-c609-4aea-b385-72e0d14076ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:42.456851 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.456814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:42.457031 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:42.456964 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:42.457031 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:42.457026 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret podName:b2143bf9-8156-489e-9d51-cdee196d4ea7 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:44.457006537 +0000 UTC m=+6.137392299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret") pod "global-pull-secret-syncer-7nlgb" (UID: "b2143bf9-8156-489e-9d51-cdee196d4ea7") : object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:42.558180 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.558140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgsj\" (UniqueName: \"kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj\") pod \"network-check-target-tq7v6\" (UID: \"783539e3-6dae-4567-ba41-f9627dddc085\") " pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:42.558379 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:42.558352 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 12:59:42.558379 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:42.558378 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 12:59:42.558507 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:42.558392 2565 projected.go:196] Error preparing data for projected volume kube-api-access-krgsj for pod openshift-network-diagnostics/network-check-target-tq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:42.558507 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:42.558447 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj podName:783539e3-6dae-4567-ba41-f9627dddc085 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:44.558429197 +0000 UTC m=+6.238814947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-krgsj" (UniqueName: "kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj") pod "network-check-target-tq7v6" (UID: "783539e3-6dae-4567-ba41-f9627dddc085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:42.820860 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.820759 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:42.821474 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:42.820899 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 12:59:42.821474 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.820950 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:42.821474 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:42.821104 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 12:59:42.839089 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.839057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4bh9v" event={"ID":"ba93a5ab-9dad-48f1-8c35-d6ce0732ac78","Type":"ContainerStarted","Data":"789236c1632a75fa46806781d301b9517ca116b12915686fe70571b520555de7"} Sep 07 12:59:42.843898 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.843872 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" event={"ID":"3af9c7e0-e966-4f88-b051-853c1e470a1b","Type":"ContainerStarted","Data":"4b0f152637f13fec4b1a563af324ac627d5655bd0d63f2352f4778bbb0540743"} Sep 07 12:59:42.845746 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.845721 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j7kfv" event={"ID":"a761ff6c-7f18-4b78-ae99-6e8d13bf1995","Type":"ContainerStarted","Data":"a58ea02f97bca5cd76c83e515e4fc43059b0e1f2d63814fe8ee3acd83710b5af"} Sep 07 12:59:42.848394 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.848328 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbjz" event={"ID":"338fa313-abb1-4bf6-8333-3e27d004296e","Type":"ContainerStarted","Data":"6ef573fb37de4a0ee060cb8296595c9aa7347913476462a82b0e5b7a5eded642"} Sep 07 12:59:42.851984 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.851941 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" event={"ID":"d0f073e8-242c-4bcf-83db-2564e842dd66","Type":"ContainerStarted","Data":"0e20d8c921ce565cf86ca165f80e94827e7056d51206f01272862b3b60c70f8a"} Sep 07 12:59:42.854285 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.854096 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" event={"ID":"62cb5867-f697-4ee1-b2e5-d2c58e33a717","Type":"ContainerStarted","Data":"4c4dcbb44cd40247a72477ff1e3d0cdf69c7e88038aa90203aee4ffd587aea4b"} Sep 07 12:59:42.856546 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.856497 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 12:54:40 +0000 UTC" deadline="2028-02-16 15:19:44.274004834 +0000 UTC" Sep 07 12:59:42.856546 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.856525 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12650h20m1.417484677s" Sep 07 12:59:42.857866 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.857807 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zbw5l" event={"ID":"96aef354-0582-4a9d-b427-4f347a152da2","Type":"ContainerStarted","Data":"b212d615d97296a4c9f4cd5ffc84138b65582c51abaaf8b7a2aed8f12d973cf5"} Sep 07 12:59:42.865086 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.865060 2565 generic.go:356] "Generic (PLEG): container finished" podID="a3fc91691ab22e8aac22ac5fac61eae7" containerID="b04b207192032515d27f0816706d62c9c81ee417260da2783f2eb4c50a744934" exitCode=0 Sep 07 12:59:42.865694 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.865672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" event={"ID":"a3fc91691ab22e8aac22ac5fac61eae7","Type":"ContainerDied","Data":"b04b207192032515d27f0816706d62c9c81ee417260da2783f2eb4c50a744934"} Sep 07 12:59:42.872665 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.872640 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-216.ec2.internal" event={"ID":"1e5b89f6dd594d68f12f82f4340ea7e5","Type":"ContainerStarted","Data":"2c76f2b6f73c3c960c4bd695f5d14702a87f0b13b4d3af72b6e636cc1d960545"} Sep 07 12:59:42.877334 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.877242 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vswfs" event={"ID":"2fddfa4d-1664-4139-8891-e7c6bc67fde4","Type":"ContainerStarted","Data":"360180427b0b9a76e1058fd549bfbc9bbafc093d9c4056be5066765383709454"} Sep 07 12:59:42.880028 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:42.879973 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hjp2h" event={"ID":"9e73d458-8cb5-4913-9ad2-54f0615ef55d","Type":"ContainerStarted","Data":"8fcf3a19354f4ed5f3bccf5ce413e977756837e66a41febdefbf7e8d82394da2"} Sep 07 12:59:43.820700 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:43.820612 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:43.820867 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:43.820744 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 12:59:43.906738 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:43.906655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" event={"ID":"a3fc91691ab22e8aac22ac5fac61eae7","Type":"ContainerStarted","Data":"bb4736d2a62370c8d7d483b0411336e424491d5ae242ba9f1722ed2a42e28e25"} Sep 07 12:59:43.931903 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:43.931769 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-216.ec2.internal" podStartSLOduration=3.931750494 podStartE2EDuration="3.931750494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:42.914264923 +0000 UTC m=+4.594650694" watchObservedRunningTime="2026-09-07 12:59:43.931750494 +0000 UTC m=+5.612136265" Sep 07 12:59:44.376642 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:44.376587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:44.376829 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:44.376803 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:44.376889 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:44.376870 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs podName:62d7435f-c609-4aea-b385-72e0d14076ba nodeName:}" failed. No retries permitted until 2026-09-07 12:59:48.376849363 +0000 UTC m=+10.057235124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs") pod "network-metrics-daemon-6wqht" (UID: "62d7435f-c609-4aea-b385-72e0d14076ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:44.477046 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:44.477010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:44.477229 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:44.477181 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:44.477299 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:44.477256 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret podName:b2143bf9-8156-489e-9d51-cdee196d4ea7 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:48.477234813 +0000 UTC m=+10.157620581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret") pod "global-pull-secret-syncer-7nlgb" (UID: "b2143bf9-8156-489e-9d51-cdee196d4ea7") : object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:44.578455 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:44.577754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgsj\" (UniqueName: \"kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj\") pod \"network-check-target-tq7v6\" (UID: \"783539e3-6dae-4567-ba41-f9627dddc085\") " pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:44.578455 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:44.577919 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 12:59:44.578455 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:44.577943 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 12:59:44.578455 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:44.577955 2565 projected.go:196] Error preparing data for projected volume kube-api-access-krgsj for pod openshift-network-diagnostics/network-check-target-tq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:44.578455 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:44.578017 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj podName:783539e3-6dae-4567-ba41-f9627dddc085 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:48.577998131 +0000 UTC m=+10.258383884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-krgsj" (UniqueName: "kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj") pod "network-check-target-tq7v6" (UID: "783539e3-6dae-4567-ba41-f9627dddc085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:44.822323 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:44.821711 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:44.822323 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:44.821834 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 12:59:44.823196 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:44.822924 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:44.823196 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:44.823035 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 12:59:45.821160 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:45.820589 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:45.821160 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:45.820768 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 12:59:46.820376 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:46.820342 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:46.820887 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:46.820476 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 12:59:46.820887 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:46.820560 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:46.820887 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:46.820680 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 12:59:47.821252 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:47.820698 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:47.821252 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:47.820841 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 12:59:48.409789 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:48.409698 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:48.409966 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:48.409869 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:48.409966 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:48.409937 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs podName:62d7435f-c609-4aea-b385-72e0d14076ba nodeName:}" failed. No retries permitted until 2026-09-07 12:59:56.409918842 +0000 UTC m=+18.090304593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs") pod "network-metrics-daemon-6wqht" (UID: "62d7435f-c609-4aea-b385-72e0d14076ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:48.510960 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:48.510918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:48.511132 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:48.511096 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:48.511206 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:48.511177 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret podName:b2143bf9-8156-489e-9d51-cdee196d4ea7 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:56.511155715 +0000 UTC m=+18.191541466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret") pod "global-pull-secret-syncer-7nlgb" (UID: "b2143bf9-8156-489e-9d51-cdee196d4ea7") : object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:48.612184 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:48.611909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgsj\" (UniqueName: \"kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj\") pod \"network-check-target-tq7v6\" (UID: \"783539e3-6dae-4567-ba41-f9627dddc085\") " pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:48.612184 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:48.612095 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 12:59:48.612184 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:48.612115 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 12:59:48.612184 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:48.612127 2565 projected.go:196] Error preparing data for projected volume kube-api-access-krgsj for pod openshift-network-diagnostics/network-check-target-tq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:48.612184 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:48.612181 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj podName:783539e3-6dae-4567-ba41-f9627dddc085 nodeName:}" failed. No retries permitted until 2026-09-07 12:59:56.612164531 +0000 UTC m=+18.292550297 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-krgsj" (UniqueName: "kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj") pod "network-check-target-tq7v6" (UID: "783539e3-6dae-4567-ba41-f9627dddc085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:48.821992 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:48.821913 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:48.822433 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:48.822060 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 12:59:48.822433 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:48.822132 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:48.822433 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:48.822236 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 12:59:49.820915 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:49.820885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:49.821094 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:49.821016 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 12:59:50.820813 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:50.820775 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:50.821279 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:50.820830 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:50.821279 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:50.820927 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 12:59:50.821279 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:50.821044 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 12:59:51.820413 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:51.820370 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:51.820569 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:51.820506 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 12:59:52.820723 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:52.820688 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:52.821182 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:52.820695 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:52.821182 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:52.820819 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 12:59:52.821182 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:52.820900 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 12:59:53.820495 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:53.820462 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:53.820700 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:53.820574 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 12:59:54.820669 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:54.820632 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:54.821094 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:54.820636 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:54.821094 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:54.820766 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 12:59:54.821094 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:54.820855 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 12:59:55.820747 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:55.820716 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:55.821102 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:55.820816 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 12:59:56.470945 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:56.470903 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:56.471131 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:56.471077 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:56.471196 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:56.471155 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs podName:62d7435f-c609-4aea-b385-72e0d14076ba nodeName:}" failed. No retries permitted until 2026-09-07 13:00:12.471134964 +0000 UTC m=+34.151520735 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs") pod "network-metrics-daemon-6wqht" (UID: "62d7435f-c609-4aea-b385-72e0d14076ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 12:59:56.571644 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:56.571608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:56.571833 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:56.571779 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:56.571892 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:56.571858 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret podName:b2143bf9-8156-489e-9d51-cdee196d4ea7 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:12.57183944 +0000 UTC m=+34.252225208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret") pod "global-pull-secret-syncer-7nlgb" (UID: "b2143bf9-8156-489e-9d51-cdee196d4ea7") : object "kube-system"/"original-pull-secret" not registered Sep 07 12:59:56.672999 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:56.672955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgsj\" (UniqueName: \"kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj\") pod \"network-check-target-tq7v6\" (UID: \"783539e3-6dae-4567-ba41-f9627dddc085\") " pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:56.673171 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:56.673088 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 12:59:56.673171 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:56.673105 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 12:59:56.673171 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:56.673115 2565 projected.go:196] Error preparing data for projected volume kube-api-access-krgsj for pod openshift-network-diagnostics/network-check-target-tq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:56.673290 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:56.673184 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj podName:783539e3-6dae-4567-ba41-f9627dddc085 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:12.673170273 +0000 UTC m=+34.353556040 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-krgsj" (UniqueName: "kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj") pod "network-check-target-tq7v6" (UID: "783539e3-6dae-4567-ba41-f9627dddc085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 12:59:56.820532 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:56.820459 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:56.820704 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:56.820579 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 12:59:56.820704 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:56.820625 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:56.820704 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:56.820696 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 12:59:57.820786 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:57.820748 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:57.821304 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:57.820882 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 12:59:58.821991 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:58.821956 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 12:59:58.822420 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:58.822059 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 12:59:58.822420 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:58.822113 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 12:59:58.822420 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:58.822203 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 12:59:59.820568 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.820274 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 12:59:59.820681 ip-10-0-142-216 kubenswrapper[2565]: E0907 12:59:59.820664 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 12:59:59.937663 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.937627 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vswfs" event={"ID":"2fddfa4d-1664-4139-8891-e7c6bc67fde4","Type":"ContainerStarted","Data":"f60c7e7eef481f91ffbdadfcb60ab82d0d83b93bf048878ea279ff48d41185f4"} Sep 07 12:59:59.939133 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.939108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hjp2h" event={"ID":"9e73d458-8cb5-4913-9ad2-54f0615ef55d","Type":"ContainerStarted","Data":"0746f79c9d62ec8ab0aee69446a5327ad721596ff22b882274fc6bca947a5d61"} Sep 07 12:59:59.940495 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.940473 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" event={"ID":"3af9c7e0-e966-4f88-b051-853c1e470a1b","Type":"ContainerStarted","Data":"6ae502be984830aed2d7239aa39596cace8f836ec26cf5497e4bc1d76fa2f871"} Sep 07 12:59:59.942056 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.942032 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j7kfv" event={"ID":"a761ff6c-7f18-4b78-ae99-6e8d13bf1995","Type":"ContainerStarted","Data":"b7829f48daed69cca6bc74450b2fbc14a7bcc8d2e8175e9b149b361c60ae6a82"} Sep 07 12:59:59.943455 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.943432 2565 generic.go:356] "Generic (PLEG): container finished" podID="338fa313-abb1-4bf6-8333-3e27d004296e" containerID="c09b7e807d574b0005baf74cab640b2b26299b9410660ed19fc3c5d357d514fa" exitCode=0 Sep 07 12:59:59.943537 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.943507 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbjz" event={"ID":"338fa313-abb1-4bf6-8333-3e27d004296e","Type":"ContainerDied","Data":"c09b7e807d574b0005baf74cab640b2b26299b9410660ed19fc3c5d357d514fa"} Sep 07 12:59:59.945178 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.945155 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" event={"ID":"d0f073e8-242c-4bcf-83db-2564e842dd66","Type":"ContainerStarted","Data":"848e9fa7f195c61c4823592312fa16fda1ff0cc244357c148754141f71dc8d0e"} Sep 07 12:59:59.948167 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.948149 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" event={"ID":"62cb5867-f697-4ee1-b2e5-d2c58e33a717","Type":"ContainerStarted","Data":"cbbcf8dbcc1617f67e57c4ba3d50500c8abc2481208f92eaf71db9768c78334b"} Sep 07 12:59:59.948329 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.948221 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" event={"ID":"62cb5867-f697-4ee1-b2e5-d2c58e33a717","Type":"ContainerStarted","Data":"8900305037e4955a0414a4e83a5d1b131e8184e1f4bee9cb6527fb8873453ab7"} Sep 07 12:59:59.948329 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.948258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" event={"ID":"62cb5867-f697-4ee1-b2e5-d2c58e33a717","Type":"ContainerStarted","Data":"1c533f0f4266e722a4239da9fa5e3b037309f693c92971ec91575b123f851a10"} Sep 07 12:59:59.948329 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.948302 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" event={"ID":"62cb5867-f697-4ee1-b2e5-d2c58e33a717","Type":"ContainerStarted","Data":"6e45c677e1a997392c625c6e6998eb689f673248cb1c54037f132a5af08433fc"} Sep 07 12:59:59.948329 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.948331 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" event={"ID":"62cb5867-f697-4ee1-b2e5-d2c58e33a717","Type":"ContainerStarted","Data":"b9146c8522af2e65c33793820994aca05c359bf86a0d29ed91ee98599523b8e0"} Sep 07 12:59:59.948478 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.948358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" event={"ID":"62cb5867-f697-4ee1-b2e5-d2c58e33a717","Type":"ContainerStarted","Data":"c89511eedae59344d955ca3933b8b15780f5578f0f86d2507c89248b314f1848"} Sep 07 12:59:59.949491 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.949471 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zbw5l" event={"ID":"96aef354-0582-4a9d-b427-4f347a152da2","Type":"ContainerStarted","Data":"5c30479233481f655c538410c63b8ad6936728de6eee8e584162305ca5f674ec"} Sep 07 12:59:59.962087 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.962050 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vswfs" podStartSLOduration=4.472583823 podStartE2EDuration="21.9620398s" totalImagesPullingTime="17.489455977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:59.960990246 +0000 UTC m=+21.641376016" watchObservedRunningTime="2026-09-07 12:59:59.9620398 +0000 UTC m=+21.642425570" Sep 07 12:59:59.962187 ip-10-0-142-216 kubenswrapper[2565]: I0907 12:59:59.962111 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-216.ec2.internal" podStartSLOduration=19.962107027 podStartE2EDuration="19.962107027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 12:59:43.932213971 +0000 UTC m=+5.612599744" watchObservedRunningTime="2026-09-07 12:59:59.962107027 +0000 UTC m=+21.642492794" Sep 07 13:00:00.038869 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.038819 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hjp2h" podStartSLOduration=4.545964586 podStartE2EDuration="22.038802487s" totalImagesPullingTime="17.492837901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:00.010236283 +0000 UTC m=+21.690622044" watchObservedRunningTime="2026-09-07 13:00:00.038802487 +0000 UTC m=+21.719188258" Sep 07 13:00:00.082069 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.081966 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bwjlq" podStartSLOduration=4.592531363 podStartE2EDuration="22.08194793s" totalImagesPullingTime="17.489416567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:00.039485191 +0000 UTC m=+21.719870974" watchObservedRunningTime="2026-09-07 13:00:00.08194793 +0000 UTC m=+21.762333701" Sep 07 13:00:00.082213 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.082161 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j7kfv" podStartSLOduration=4.532919677 podStartE2EDuration="22.082149539s" totalImagesPullingTime="17.549229862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:00.080010358 +0000 UTC m=+21.760396128" watchObservedRunningTime="2026-09-07 13:00:00.082149539 +0000 UTC m=+21.762535310" Sep 07 13:00:00.104961 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.104920 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zbw5l" podStartSLOduration=4.60709637 podStartE2EDuration="22.104906982s" totalImagesPullingTime="17.497810612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:00.10433006 +0000 UTC m=+21.784715833" watchObservedRunningTime="2026-09-07 13:00:00.104906982 +0000 UTC m=+21.785292751" Sep 07 13:00:00.600202 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.600179 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 07 13:00:00.790296 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.790109 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T13:00:00.600198325Z","UUID":"f4533489-91f9-4a17-a678-a5ef5cf9c433","Handler":null,"Name":"","Endpoint":""} Sep 07 13:00:00.791632 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.791588 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 07 13:00:00.791632 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.791632 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 07 13:00:00.820808 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.820784 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:00.820951 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:00.820899 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 13:00:00.820951 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.820784 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:00.821073 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:00.820998 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 13:00:00.953234 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.953192 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4bh9v" event={"ID":"ba93a5ab-9dad-48f1-8c35-d6ce0732ac78","Type":"ContainerStarted","Data":"6d9901013be23bbd60d97d7ec1cca913d89b82e22712fd8039bc1beff6d62fe8"} Sep 07 13:00:00.955365 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.955339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" event={"ID":"3af9c7e0-e966-4f88-b051-853c1e470a1b","Type":"ContainerStarted","Data":"cc8fd1f0227b6a16ccf36b6b79d7d790a0d93f03d9b3e5df7de78869c6ae8253"} Sep 07 13:00:00.972803 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:00.972756 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4bh9v" podStartSLOduration=5.480922465 podStartE2EDuration="22.972742047s" totalImagesPullingTime="17.491819582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:00.972641203 +0000 UTC m=+22.653026972" watchObservedRunningTime="2026-09-07 13:00:00.972742047 +0000 UTC m=+22.653127816" Sep 07 13:00:01.820750 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:01.820718 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:01.820932 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:01.820851 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 13:00:02.245381 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:02.245352 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zbw5l" Sep 07 13:00:02.246168 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:02.246145 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zbw5l" Sep 07 13:00:02.821398 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:02.821319 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:02.821549 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:02.821425 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 13:00:02.821549 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:02.821475 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:02.821710 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:02.821571 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 13:00:02.963091 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:02.963051 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" event={"ID":"3af9c7e0-e966-4f88-b051-853c1e470a1b","Type":"ContainerStarted","Data":"738fea75d3861172b4b2ca7bac1e6fdf609c4161a8f540f84aa0cdf84d97df00"} Sep 07 13:00:02.967107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:02.967063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" event={"ID":"62cb5867-f697-4ee1-b2e5-d2c58e33a717","Type":"ContainerStarted","Data":"9ea8731a4ed4f50a80a6397f9bfdf9d5aac49645a1017bc4c09373474222b9f9"} Sep 07 13:00:02.986532 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:02.986487 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tclkp" podStartSLOduration=4.670392727 podStartE2EDuration="24.986469888s" totalImagesPullingTime="20.316077161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:02.985950597 +0000 UTC m=+24.666336378" watchObservedRunningTime="2026-09-07 13:00:02.986469888 +0000 UTC m=+24.666855661" Sep 07 13:00:03.820613 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:03.820432 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:03.821019 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:03.820697 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 13:00:04.821273 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:04.821203 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:04.821273 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:04.821243 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:04.821965 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:04.821323 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 13:00:04.821965 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:04.821437 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 13:00:04.972858 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:04.972825 2565 generic.go:356] "Generic (PLEG): container finished" podID="338fa313-abb1-4bf6-8333-3e27d004296e" containerID="09d319fb7f060e916b8a6f524c7a2e9d913ad52c829dbc008708f495ccb83505" exitCode=0 Sep 07 13:00:04.973039 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:04.972908 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbjz" event={"ID":"338fa313-abb1-4bf6-8333-3e27d004296e","Type":"ContainerDied","Data":"09d319fb7f060e916b8a6f524c7a2e9d913ad52c829dbc008708f495ccb83505"} Sep 07 13:00:04.976638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:04.976613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" event={"ID":"62cb5867-f697-4ee1-b2e5-d2c58e33a717","Type":"ContainerStarted","Data":"e1229effb172b249f2717481248daf2ec1bdaf5dfa953af7c82337f65e0d737e"} Sep 07 13:00:04.976928 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:04.976907 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 13:00:04.991888 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:04.991865 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 13:00:05.820746 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:05.820715 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:05.820955 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:05.820815 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 13:00:05.979173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:05.979017 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 13:00:05.979504 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:05.979223 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 13:00:05.993782 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:05.993757 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 13:00:06.031756 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:06.031701 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" podStartSLOduration=9.481393561 podStartE2EDuration="27.031682632s" totalImagesPullingTime="17.550289071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:05.042208623 +0000 UTC m=+26.722594392" watchObservedRunningTime="2026-09-07 13:00:06.031682632 +0000 UTC m=+27.712068402" Sep 07 13:00:06.394450 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:06.394421 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6wqht"] Sep 07 13:00:06.394604 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:06.394532 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:06.394658 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:06.394639 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 13:00:06.397254 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:06.397232 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7nlgb"] Sep 07 13:00:06.397376 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:06.397321 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:06.397437 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:06.397417 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 13:00:06.416968 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:06.416901 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tq7v6"] Sep 07 13:00:06.417083 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:06.417005 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:06.417127 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:06.417075 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 13:00:06.983114 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:06.983022 2565 generic.go:356] "Generic (PLEG): container finished" podID="338fa313-abb1-4bf6-8333-3e27d004296e" containerID="c1d508eb7f9c8ca047e7c0785b38334a0a48d625f6b264aefd0e7dd85e7f8a53" exitCode=0 Sep 07 13:00:06.983114 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:06.983100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbjz" event={"ID":"338fa313-abb1-4bf6-8333-3e27d004296e","Type":"ContainerDied","Data":"c1d508eb7f9c8ca047e7c0785b38334a0a48d625f6b264aefd0e7dd85e7f8a53"} Sep 07 13:00:07.820274 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:07.820190 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:07.820410 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:07.820290 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 13:00:07.820410 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:07.820345 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:07.820501 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:07.820427 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 13:00:08.821028 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:08.821005 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:08.821464 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:08.821091 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 13:00:08.989651 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:08.989590 2565 generic.go:356] "Generic (PLEG): container finished" podID="338fa313-abb1-4bf6-8333-3e27d004296e" containerID="fd6cf4f7a57837ac32112893ec559ac9e34a331978d5c9676047afa62e2079ee" exitCode=0 Sep 07 13:00:08.989651 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:08.989628 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbjz" event={"ID":"338fa313-abb1-4bf6-8333-3e27d004296e","Type":"ContainerDied","Data":"fd6cf4f7a57837ac32112893ec559ac9e34a331978d5c9676047afa62e2079ee"} Sep 07 13:00:09.820373 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:09.820343 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:09.820512 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:09.820344 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:09.820512 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:09.820440 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 13:00:09.820628 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:09.820506 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 13:00:10.080645 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:10.080557 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zbw5l" Sep 07 13:00:10.081057 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:10.080705 2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 07 13:00:10.081160 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:10.081142 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zbw5l" Sep 07 13:00:10.820634 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:10.820584 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:10.820829 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:10.820742 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 13:00:11.820468 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:11.820438 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:11.820996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:11.820479 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:11.820996 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:11.820558 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tq7v6" podUID="783539e3-6dae-4567-ba41-f9627dddc085" Sep 07 13:00:11.820996 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:11.820706 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7nlgb" podUID="b2143bf9-8156-489e-9d51-cdee196d4ea7" Sep 07 13:00:12.497692 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:12.497658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:12.497851 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:12.497818 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 13:00:12.497910 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:12.497898 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs podName:62d7435f-c609-4aea-b385-72e0d14076ba nodeName:}" failed. No retries permitted until 2026-09-07 13:00:44.497876604 +0000 UTC m=+66.178262364 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs") pod "network-metrics-daemon-6wqht" (UID: "62d7435f-c609-4aea-b385-72e0d14076ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 13:00:12.598160 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:12.598127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:12.598324 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:12.598246 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 13:00:12.598324 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:12.598303 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret podName:b2143bf9-8156-489e-9d51-cdee196d4ea7 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:44.598289112 +0000 UTC m=+66.278674859 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret") pod "global-pull-secret-syncer-7nlgb" (UID: "b2143bf9-8156-489e-9d51-cdee196d4ea7") : object "kube-system"/"original-pull-secret" not registered Sep 07 13:00:12.699085 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:12.699000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgsj\" (UniqueName: \"kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj\") pod \"network-check-target-tq7v6\" (UID: \"783539e3-6dae-4567-ba41-f9627dddc085\") " pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:12.699249 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:12.699201 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 13:00:12.699249 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:12.699229 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 13:00:12.699249 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:12.699242 2565 projected.go:196] Error preparing data for projected volume kube-api-access-krgsj for pod openshift-network-diagnostics/network-check-target-tq7v6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 13:00:12.699411 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:12.699295 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj podName:783539e3-6dae-4567-ba41-f9627dddc085 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:44.6992796 +0000 UTC m=+66.379665375 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-krgsj" (UniqueName: "kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj") pod "network-check-target-tq7v6" (UID: "783539e3-6dae-4567-ba41-f9627dddc085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 13:00:12.820854 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:12.820819 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:12.821253 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:12.820965 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wqht" podUID="62d7435f-c609-4aea-b385-72e0d14076ba" Sep 07 13:00:13.163488 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.163409 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-216.ec2.internal" event="NodeReady" Sep 07 13:00:13.163671 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.163557 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 07 13:00:13.215277 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.215251 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xg2sx"] Sep 07 13:00:13.235972 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.235944 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2p52x"] Sep 07 13:00:13.236113 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.236072 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.240454 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.240428 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 07 13:00:13.240590 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.240529 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rqgx2\"" Sep 07 13:00:13.240722 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.240703 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 07 13:00:13.251999 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.251979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xg2sx"] Sep 07 13:00:13.252095 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.252022 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2p52x"] Sep 07 13:00:13.252095 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.252091 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2p52x" Sep 07 13:00:13.255137 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.255121 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 07 13:00:13.255137 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.255133 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 07 13:00:13.255293 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.255191 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n5q6z\"" Sep 07 13:00:13.255439 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.255419 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 07 13:00:13.404274 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.404163 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65861502-da2f-4c4d-a629-c860d4c3f6e9-config-volume\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.404274 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.404224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d48d2f85-2ba1-4b88-9f3d-f512294a07f3-cert\") pod \"ingress-canary-2p52x\" (UID: \"d48d2f85-2ba1-4b88-9f3d-f512294a07f3\") " pod="openshift-ingress-canary/ingress-canary-2p52x" Sep 07 13:00:13.404274 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.404257 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2s2p\" (UniqueName: \"kubernetes.io/projected/d48d2f85-2ba1-4b88-9f3d-f512294a07f3-kube-api-access-w2s2p\") pod \"ingress-canary-2p52x\" (UID: \"d48d2f85-2ba1-4b88-9f3d-f512294a07f3\") " pod="openshift-ingress-canary/ingress-canary-2p52x" Sep 07 13:00:13.404554 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.404338 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.404554 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.404363 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65861502-da2f-4c4d-a629-c860d4c3f6e9-tmp-dir\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.404554 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.404402 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmd96\" (UniqueName: \"kubernetes.io/projected/65861502-da2f-4c4d-a629-c860d4c3f6e9-kube-api-access-nmd96\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.505501 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.505422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.505501 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.505463 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65861502-da2f-4c4d-a629-c860d4c3f6e9-tmp-dir\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.505735 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.505493 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmd96\" (UniqueName: \"kubernetes.io/projected/65861502-da2f-4c4d-a629-c860d4c3f6e9-kube-api-access-nmd96\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.505735 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:13.505632 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 13:00:13.505735 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.505642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65861502-da2f-4c4d-a629-c860d4c3f6e9-config-volume\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.505735 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.505679 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d48d2f85-2ba1-4b88-9f3d-f512294a07f3-cert\") pod \"ingress-canary-2p52x\" (UID: \"d48d2f85-2ba1-4b88-9f3d-f512294a07f3\") " pod="openshift-ingress-canary/ingress-canary-2p52x" Sep 07 13:00:13.505735 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:13.505691 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls podName:65861502-da2f-4c4d-a629-c860d4c3f6e9 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:14.005673544 +0000 UTC m=+35.686059297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls") pod "dns-default-xg2sx" (UID: "65861502-da2f-4c4d-a629-c860d4c3f6e9") : secret "dns-default-metrics-tls" not found Sep 07 13:00:13.505735 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.505717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2s2p\" (UniqueName: \"kubernetes.io/projected/d48d2f85-2ba1-4b88-9f3d-f512294a07f3-kube-api-access-w2s2p\") pod \"ingress-canary-2p52x\" (UID: \"d48d2f85-2ba1-4b88-9f3d-f512294a07f3\") " pod="openshift-ingress-canary/ingress-canary-2p52x" Sep 07 13:00:13.506091 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.505750 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65861502-da2f-4c4d-a629-c860d4c3f6e9-tmp-dir\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.506091 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.506079 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65861502-da2f-4c4d-a629-c860d4c3f6e9-config-volume\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.509445 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.509420 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d48d2f85-2ba1-4b88-9f3d-f512294a07f3-cert\") pod \"ingress-canary-2p52x\" (UID: \"d48d2f85-2ba1-4b88-9f3d-f512294a07f3\") " pod="openshift-ingress-canary/ingress-canary-2p52x" Sep 07 13:00:13.520849 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.520822 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2s2p\" (UniqueName: \"kubernetes.io/projected/d48d2f85-2ba1-4b88-9f3d-f512294a07f3-kube-api-access-w2s2p\") pod \"ingress-canary-2p52x\" (UID: \"d48d2f85-2ba1-4b88-9f3d-f512294a07f3\") " pod="openshift-ingress-canary/ingress-canary-2p52x" Sep 07 13:00:13.529188 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.529132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmd96\" (UniqueName: \"kubernetes.io/projected/65861502-da2f-4c4d-a629-c860d4c3f6e9-kube-api-access-nmd96\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:13.561114 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.561066 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2p52x" Sep 07 13:00:13.820749 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.820672 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:13.820919 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.820672 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:13.823464 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.823411 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 07 13:00:13.823464 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.823437 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 13:00:13.824084 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.824066 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8d5qt\"" Sep 07 13:00:13.824193 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.824113 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 07 13:00:13.887192 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.887163 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn"] Sep 07 13:00:13.905887 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.905862 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:13.913029 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.912851 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 07 13:00:13.913029 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.912878 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 07 13:00:13.913916 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.913900 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 07 13:00:13.916231 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.916216 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 07 13:00:13.920410 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.920367 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pp6wd\"" Sep 07 13:00:13.920866 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:13.920838 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn"] Sep 07 13:00:14.000819 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.000784 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-kwlqf"] Sep 07 13:00:14.009794 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.009764 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:14.009944 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.009808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:14.009944 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.009900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ab8a00e-df57-45f1-b68c-2198e83caf41-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:14.010063 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.009940 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 13:00:14.010063 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.009958 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdq62\" (UniqueName: \"kubernetes.io/projected/1ab8a00e-df57-45f1-b68c-2198e83caf41-kube-api-access-jdq62\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:14.010063 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.010017 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls podName:65861502-da2f-4c4d-a629-c860d4c3f6e9 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:15.009994368 +0000 UTC m=+36.690380118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls") pod "dns-default-xg2sx" (UID: "65861502-da2f-4c4d-a629-c860d4c3f6e9") : secret "dns-default-metrics-tls" not found Sep 07 13:00:14.018576 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.018549 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-kwlqf"] Sep 07 13:00:14.018703 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.018658 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:14.022925 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.022905 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 07 13:00:14.023046 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.022948 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kcxq5\"" Sep 07 13:00:14.026100 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.026079 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 07 13:00:14.093327 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.093241 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-696c9f54dc-grjtd"] Sep 07 13:00:14.109845 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.109796 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.110065 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.110052 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-696c9f54dc-grjtd"] Sep 07 13:00:14.110438 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.110404 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ab8a00e-df57-45f1-b68c-2198e83caf41-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:14.110542 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.110447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdq62\" (UniqueName: \"kubernetes.io/projected/1ab8a00e-df57-45f1-b68c-2198e83caf41-kube-api-access-jdq62\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:14.110542 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.110506 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:14.110691 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.110660 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:14.110744 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.110728 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls podName:1ab8a00e-df57-45f1-b68c-2198e83caf41 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:14.610710698 +0000 UTC m=+36.291096450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q8dpn" (UID: "1ab8a00e-df57-45f1-b68c-2198e83caf41") : secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:14.111149 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.111127 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ab8a00e-df57-45f1-b68c-2198e83caf41-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:14.113912 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.113885 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 07 13:00:14.114061 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.113882 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 07 13:00:14.114723 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.114645 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dn6gs\"" Sep 07 13:00:14.115324 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.115307 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 07 13:00:14.122662 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.122638 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 07 13:00:14.125898 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.125856 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdq62\" (UniqueName: \"kubernetes.io/projected/1ab8a00e-df57-45f1-b68c-2198e83caf41-kube-api-access-jdq62\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:14.211658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.211615 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-image-registry-private-configuration\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.211824 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.211669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.211824 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.211756 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-trusted-ca\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.211824 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.211781 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrj6\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-kube-api-access-dqrj6\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.211963 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.211832 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4212367c-1066-4682-859a-0abda62d959a-ca-trust-extracted\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.211963 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.211921 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:14.211963 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.211956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-registry-certificates\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.212112 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.211989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-bound-sa-token\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.212112 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.212094 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/974b5145-6856-4fea-b95d-e393bd76baed-nginx-conf\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:14.212214 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.212131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-installation-pull-secrets\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.313096 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313057 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4212367c-1066-4682-859a-0abda62d959a-ca-trust-extracted\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.313286 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:14.313286 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.313220 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 13:00:14.313286 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-registry-certificates\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.313286 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-bound-sa-token\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.313494 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.313297 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert podName:974b5145-6856-4fea-b95d-e393bd76baed nodeName:}" failed. No retries permitted until 2026-09-07 13:00:14.813275265 +0000 UTC m=+36.493661013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-kwlqf" (UID: "974b5145-6856-4fea-b95d-e393bd76baed") : secret "networking-console-plugin-cert" not found Sep 07 13:00:14.313494 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313346 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/974b5145-6856-4fea-b95d-e393bd76baed-nginx-conf\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:14.313494 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313374 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-installation-pull-secrets\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.313494 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313407 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-image-registry-private-configuration\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.313494 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313441 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.313827 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-trusted-ca\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.313827 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4212367c-1066-4682-859a-0abda62d959a-ca-trust-extracted\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.313827 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrj6\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-kube-api-access-dqrj6\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.313827 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.313719 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 13:00:14.313827 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.313734 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-696c9f54dc-grjtd: secret "image-registry-tls" not found Sep 07 13:00:14.313827 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.313791 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls podName:4212367c-1066-4682-859a-0abda62d959a nodeName:}" failed. No retries permitted until 2026-09-07 13:00:14.813775829 +0000 UTC m=+36.494161598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls") pod "image-registry-696c9f54dc-grjtd" (UID: "4212367c-1066-4682-859a-0abda62d959a") : secret "image-registry-tls" not found Sep 07 13:00:14.314142 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.313920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-registry-certificates\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.314142 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.314118 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/974b5145-6856-4fea-b95d-e393bd76baed-nginx-conf\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:14.314635 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.314589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-trusted-ca\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.316154 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.316132 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-installation-pull-secrets\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.316354 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.316333 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-image-registry-private-configuration\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.322092 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.322069 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-bound-sa-token\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.322291 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.322270 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrj6\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-kube-api-access-dqrj6\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.616511 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.616471 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:14.616737 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.616715 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:14.616874 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.616791 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls podName:1ab8a00e-df57-45f1-b68c-2198e83caf41 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:15.616771798 +0000 UTC m=+37.297157547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q8dpn" (UID: "1ab8a00e-df57-45f1-b68c-2198e83caf41") : secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:14.818457 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.818418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:14.818657 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.818525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:14.818657 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.818588 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 13:00:14.818657 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.818628 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-696c9f54dc-grjtd: secret "image-registry-tls" not found Sep 07 13:00:14.818814 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.818656 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 13:00:14.818814 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.818696 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls podName:4212367c-1066-4682-859a-0abda62d959a nodeName:}" failed. No retries permitted until 2026-09-07 13:00:15.818675002 +0000 UTC m=+37.499060781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls") pod "image-registry-696c9f54dc-grjtd" (UID: "4212367c-1066-4682-859a-0abda62d959a") : secret "image-registry-tls" not found Sep 07 13:00:14.818814 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:14.818724 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert podName:974b5145-6856-4fea-b95d-e393bd76baed nodeName:}" failed. No retries permitted until 2026-09-07 13:00:15.81870503 +0000 UTC m=+37.499090804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-kwlqf" (UID: "974b5145-6856-4fea-b95d-e393bd76baed") : secret "networking-console-plugin-cert" not found Sep 07 13:00:14.824668 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.824643 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:14.826857 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.826822 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 13:00:14.826977 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:14.826927 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5v9xw\"" Sep 07 13:00:15.020423 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:15.020357 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:15.020559 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:15.020505 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 13:00:15.020620 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:15.020564 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls podName:65861502-da2f-4c4d-a629-c860d4c3f6e9 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:17.020547566 +0000 UTC m=+38.700933314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls") pod "dns-default-xg2sx" (UID: "65861502-da2f-4c4d-a629-c860d4c3f6e9") : secret "dns-default-metrics-tls" not found Sep 07 13:00:15.343178 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:15.342956 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2p52x"] Sep 07 13:00:15.348638 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:15.348608 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48d2f85_2ba1_4b88_9f3d_f512294a07f3.slice/crio-8f4e0dde608391611bb9dbab54e9b00023fb521a8e73f07f6b7e6c5370c094ca WatchSource:0}: Error finding container 8f4e0dde608391611bb9dbab54e9b00023fb521a8e73f07f6b7e6c5370c094ca: Status 404 returned error can't find the container with id 8f4e0dde608391611bb9dbab54e9b00023fb521a8e73f07f6b7e6c5370c094ca Sep 07 13:00:15.624692 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:15.624579 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:15.624811 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:15.624721 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:15.624811 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:15.624785 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls podName:1ab8a00e-df57-45f1-b68c-2198e83caf41 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:17.624769336 +0000 UTC m=+39.305155083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q8dpn" (UID: "1ab8a00e-df57-45f1-b68c-2198e83caf41") : secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:15.826406 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:15.826371 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:15.826886 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:15.826453 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:15.826886 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:15.826528 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 13:00:15.826886 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:15.826548 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 13:00:15.826886 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:15.826617 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert podName:974b5145-6856-4fea-b95d-e393bd76baed nodeName:}" failed. No retries permitted until 2026-09-07 13:00:17.826582863 +0000 UTC m=+39.506968611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-kwlqf" (UID: "974b5145-6856-4fea-b95d-e393bd76baed") : secret "networking-console-plugin-cert" not found Sep 07 13:00:15.826886 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:15.826551 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-696c9f54dc-grjtd: secret "image-registry-tls" not found Sep 07 13:00:15.826886 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:15.826673 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls podName:4212367c-1066-4682-859a-0abda62d959a nodeName:}" failed. No retries permitted until 2026-09-07 13:00:17.826659528 +0000 UTC m=+39.507045278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls") pod "image-registry-696c9f54dc-grjtd" (UID: "4212367c-1066-4682-859a-0abda62d959a") : secret "image-registry-tls" not found Sep 07 13:00:16.009323 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:16.009290 2565 generic.go:356] "Generic (PLEG): container finished" podID="338fa313-abb1-4bf6-8333-3e27d004296e" containerID="41ba9954a0a20a384c660df2e7358c2cf97bc9a8fa539753893a4aeef06ce98a" exitCode=0 Sep 07 13:00:16.009494 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:16.009374 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbjz" event={"ID":"338fa313-abb1-4bf6-8333-3e27d004296e","Type":"ContainerDied","Data":"41ba9954a0a20a384c660df2e7358c2cf97bc9a8fa539753893a4aeef06ce98a"} Sep 07 13:00:16.010635 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:16.010612 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2p52x" event={"ID":"d48d2f85-2ba1-4b88-9f3d-f512294a07f3","Type":"ContainerStarted","Data":"8f4e0dde608391611bb9dbab54e9b00023fb521a8e73f07f6b7e6c5370c094ca"} Sep 07 13:00:17.016738 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:17.016698 2565 generic.go:356] "Generic (PLEG): container finished" podID="338fa313-abb1-4bf6-8333-3e27d004296e" containerID="4ae35b924056f14026e54e1bb61821bb9ed199d96c8852b0d460d29ffd174916" exitCode=0 Sep 07 13:00:17.017235 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:17.016764 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbjz" event={"ID":"338fa313-abb1-4bf6-8333-3e27d004296e","Type":"ContainerDied","Data":"4ae35b924056f14026e54e1bb61821bb9ed199d96c8852b0d460d29ffd174916"} Sep 07 13:00:17.037040 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:17.037000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:17.037284 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:17.037212 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 13:00:17.037284 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:17.037272 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls podName:65861502-da2f-4c4d-a629-c860d4c3f6e9 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:21.037251103 +0000 UTC m=+42.717636865 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls") pod "dns-default-xg2sx" (UID: "65861502-da2f-4c4d-a629-c860d4c3f6e9") : secret "dns-default-metrics-tls" not found Sep 07 13:00:17.641829 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:17.641789 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:17.641999 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:17.641941 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:17.642046 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:17.642008 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls podName:1ab8a00e-df57-45f1-b68c-2198e83caf41 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:21.641992053 +0000 UTC m=+43.322377801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q8dpn" (UID: "1ab8a00e-df57-45f1-b68c-2198e83caf41") : secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:17.843786 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:17.843741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:17.844032 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:17.843884 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 13:00:17.844032 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:17.843905 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-696c9f54dc-grjtd: secret "image-registry-tls" not found Sep 07 13:00:17.844032 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:17.843924 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:17.844032 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:17.843981 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls podName:4212367c-1066-4682-859a-0abda62d959a nodeName:}" failed. No retries permitted until 2026-09-07 13:00:21.843962503 +0000 UTC m=+43.524348274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls") pod "image-registry-696c9f54dc-grjtd" (UID: "4212367c-1066-4682-859a-0abda62d959a") : secret "image-registry-tls" not found Sep 07 13:00:17.844032 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:17.844032 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 13:00:17.844323 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:17.844087 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert podName:974b5145-6856-4fea-b95d-e393bd76baed nodeName:}" failed. No retries permitted until 2026-09-07 13:00:21.844071805 +0000 UTC m=+43.524457553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-kwlqf" (UID: "974b5145-6856-4fea-b95d-e393bd76baed") : secret "networking-console-plugin-cert" not found Sep 07 13:00:18.023681 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:18.023583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjbjz" event={"ID":"338fa313-abb1-4bf6-8333-3e27d004296e","Type":"ContainerStarted","Data":"c0b235a85809a66cbfcea241f694b1866b6b313c3a2d0d56cb2c2a25ed89f3bf"} Sep 07 13:00:18.046361 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:18.046316 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fjbjz" podStartSLOduration=7.57557628 podStartE2EDuration="39.046295806s" totalImagesPullingTime="31.470719526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:18.045727445 +0000 UTC m=+39.726113218" watchObservedRunningTime="2026-09-07 13:00:18.046295806 +0000 UTC m=+39.726681586" Sep 07 13:00:19.028277 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.028242 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2p52x" event={"ID":"d48d2f85-2ba1-4b88-9f3d-f512294a07f3","Type":"ContainerStarted","Data":"7f666d1b33b5ccaea72cae4ebc16681b6865afdd10bbe518ce2791c87d2f67b6"} Sep 07 13:00:19.045393 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.045342 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2p52x" podStartSLOduration=2.5709955239999998 podStartE2EDuration="6.045325617s" totalImagesPullingTime="3.474330093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:19.044172032 +0000 UTC m=+40.724557801" watchObservedRunningTime="2026-09-07 13:00:19.045325617 +0000 UTC m=+40.725711387" Sep 07 13:00:19.319893 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.319811 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb"] Sep 07 13:00:19.345867 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.345837 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb"] Sep 07 13:00:19.346035 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.345946 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb" Sep 07 13:00:19.348270 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.348244 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 07 13:00:19.348623 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.348582 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 07 13:00:19.348752 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.348733 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-czmdf\"" Sep 07 13:00:19.354147 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.354128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6t6\" (UniqueName: \"kubernetes.io/projected/a53dca13-f533-42fe-b3ba-99671f656e9d-kube-api-access-sc6t6\") pod \"migrator-fdbb8fdff-xd8sb\" (UID: \"a53dca13-f533-42fe-b3ba-99671f656e9d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb" Sep 07 13:00:19.454685 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.454649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6t6\" (UniqueName: \"kubernetes.io/projected/a53dca13-f533-42fe-b3ba-99671f656e9d-kube-api-access-sc6t6\") pod \"migrator-fdbb8fdff-xd8sb\" (UID: \"a53dca13-f533-42fe-b3ba-99671f656e9d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb" Sep 07 13:00:19.464373 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.464345 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6t6\" (UniqueName: \"kubernetes.io/projected/a53dca13-f533-42fe-b3ba-99671f656e9d-kube-api-access-sc6t6\") pod \"migrator-fdbb8fdff-xd8sb\" (UID: \"a53dca13-f533-42fe-b3ba-99671f656e9d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb" Sep 07 13:00:19.654035 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.653951 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb" Sep 07 13:00:19.934846 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:19.934780 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb"] Sep 07 13:00:19.937711 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:19.937679 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53dca13_f533_42fe_b3ba_99671f656e9d.slice/crio-344b13b812e186f5b63c9d2479d3c1fa02f5eaab1a3a7f1657ad2f17cec9d6e0 WatchSource:0}: Error finding container 344b13b812e186f5b63c9d2479d3c1fa02f5eaab1a3a7f1657ad2f17cec9d6e0: Status 404 returned error can't find the container with id 344b13b812e186f5b63c9d2479d3c1fa02f5eaab1a3a7f1657ad2f17cec9d6e0 Sep 07 13:00:20.031379 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:20.031343 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb" event={"ID":"a53dca13-f533-42fe-b3ba-99671f656e9d","Type":"ContainerStarted","Data":"344b13b812e186f5b63c9d2479d3c1fa02f5eaab1a3a7f1657ad2f17cec9d6e0"} Sep 07 13:00:21.067840 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.067807 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:21.068186 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.067964 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 07 13:00:21.068186 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.068027 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls podName:65861502-da2f-4c4d-a629-c860d4c3f6e9 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:29.068012739 +0000 UTC m=+50.748398487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls") pod "dns-default-xg2sx" (UID: "65861502-da2f-4c4d-a629-c860d4c3f6e9") : secret "dns-default-metrics-tls" not found Sep 07 13:00:21.673198 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.673156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:21.673371 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.673273 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:21.673371 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.673326 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls podName:1ab8a00e-df57-45f1-b68c-2198e83caf41 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:29.673312453 +0000 UTC m=+51.353698214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q8dpn" (UID: "1ab8a00e-df57-45f1-b68c-2198e83caf41") : secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:21.780342 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.780310 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wj79l"] Sep 07 13:00:21.798139 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.798103 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wj79l"] Sep 07 13:00:21.798285 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.798224 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.801404 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.801378 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 07 13:00:21.801548 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.801529 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 07 13:00:21.801909 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.801893 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 07 13:00:21.802160 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.802142 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 07 13:00:21.802626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.802609 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9l2d6\"" Sep 07 13:00:21.874346 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.874322 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/852d1deb-e626-454d-a9b0-19759119b16d-data-volume\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.874483 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.874351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.874483 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.874374 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:21.874483 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.874391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.874483 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.874481 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 13:00:21.874636 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.874537 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqxg\" (UniqueName: \"kubernetes.io/projected/852d1deb-e626-454d-a9b0-19759119b16d-kube-api-access-sdqxg\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.874636 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.874559 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert podName:974b5145-6856-4fea-b95d-e393bd76baed nodeName:}" failed. No retries permitted until 2026-09-07 13:00:29.874537161 +0000 UTC m=+51.554922913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-kwlqf" (UID: "974b5145-6856-4fea-b95d-e393bd76baed") : secret "networking-console-plugin-cert" not found Sep 07 13:00:21.874636 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.874629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:21.874732 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.874653 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852d1deb-e626-454d-a9b0-19759119b16d-crio-socket\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.874765 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.874740 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 07 13:00:21.874765 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.874751 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-696c9f54dc-grjtd: secret "image-registry-tls" not found Sep 07 13:00:21.874823 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.874795 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls podName:4212367c-1066-4682-859a-0abda62d959a nodeName:}" failed. No retries permitted until 2026-09-07 13:00:29.874782351 +0000 UTC m=+51.555168100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls") pod "image-registry-696c9f54dc-grjtd" (UID: "4212367c-1066-4682-859a-0abda62d959a") : secret "image-registry-tls" not found Sep 07 13:00:21.975686 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.975664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/852d1deb-e626-454d-a9b0-19759119b16d-data-volume\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.975788 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.975692 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.975788 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.975718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.975858 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.975805 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 13:00:21.975895 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.975854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqxg\" (UniqueName: \"kubernetes.io/projected/852d1deb-e626-454d-a9b0-19759119b16d-kube-api-access-sdqxg\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.975895 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.975875 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm podName:852d1deb-e626-454d-a9b0-19759119b16d nodeName:}" failed. No retries permitted until 2026-09-07 13:00:22.475855977 +0000 UTC m=+44.156241737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wj79l" (UID: "852d1deb-e626-454d-a9b0-19759119b16d") : configmap "kube-rbac-proxy" not found Sep 07 13:00:21.975976 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.975809 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 13:00:21.975976 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.975940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852d1deb-e626-454d-a9b0-19759119b16d-crio-socket\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.975976 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:21.975948 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls podName:852d1deb-e626-454d-a9b0-19759119b16d nodeName:}" failed. No retries permitted until 2026-09-07 13:00:22.475935714 +0000 UTC m=+44.156321466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wj79l" (UID: "852d1deb-e626-454d-a9b0-19759119b16d") : secret "insights-runtime-extractor-tls" not found Sep 07 13:00:21.976079 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.976015 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/852d1deb-e626-454d-a9b0-19759119b16d-data-volume\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.976079 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.976064 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852d1deb-e626-454d-a9b0-19759119b16d-crio-socket\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:21.985681 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:21.985654 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqxg\" (UniqueName: \"kubernetes.io/projected/852d1deb-e626-454d-a9b0-19759119b16d-kube-api-access-sdqxg\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:22.038708 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:22.038667 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb" event={"ID":"a53dca13-f533-42fe-b3ba-99671f656e9d","Type":"ContainerStarted","Data":"841f34ee657bed70d427517b9fba09c6bafe16daa2f99c4eac97abde4aabc3a8"} Sep 07 13:00:22.038879 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:22.038727 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb" event={"ID":"a53dca13-f533-42fe-b3ba-99671f656e9d","Type":"ContainerStarted","Data":"eba729184af8e7a5deb1a92f6bd2ce0764cbebe86720865cf66e1b8eaba37a5e"} Sep 07 13:00:22.056669 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:22.056620 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xd8sb" podStartSLOduration=1.179875343 podStartE2EDuration="3.056588682s" totalImagesPullingTime="1.876713339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:22.056371485 +0000 UTC m=+43.736757255" watchObservedRunningTime="2026-09-07 13:00:22.056588682 +0000 UTC m=+43.736974490" Sep 07 13:00:22.483761 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:22.481123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:22.483761 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:22.481338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:22.483761 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:22.481490 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 13:00:22.483761 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:22.481587 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm podName:852d1deb-e626-454d-a9b0-19759119b16d nodeName:}" failed. No retries permitted until 2026-09-07 13:00:23.481549442 +0000 UTC m=+45.161935205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wj79l" (UID: "852d1deb-e626-454d-a9b0-19759119b16d") : configmap "kube-rbac-proxy" not found Sep 07 13:00:22.483761 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:22.481711 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 13:00:22.483761 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:22.481754 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls podName:852d1deb-e626-454d-a9b0-19759119b16d nodeName:}" failed. No retries permitted until 2026-09-07 13:00:23.481742732 +0000 UTC m=+45.162128497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wj79l" (UID: "852d1deb-e626-454d-a9b0-19759119b16d") : secret "insights-runtime-extractor-tls" not found Sep 07 13:00:23.491181 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:23.491135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:23.491181 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:23.491187 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:23.491618 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:23.491254 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 13:00:23.491618 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:23.491310 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 13:00:23.491618 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:23.491329 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm podName:852d1deb-e626-454d-a9b0-19759119b16d nodeName:}" failed. No retries permitted until 2026-09-07 13:00:25.49131444 +0000 UTC m=+47.171700188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wj79l" (UID: "852d1deb-e626-454d-a9b0-19759119b16d") : configmap "kube-rbac-proxy" not found Sep 07 13:00:23.491618 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:23.491360 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls podName:852d1deb-e626-454d-a9b0-19759119b16d nodeName:}" failed. No retries permitted until 2026-09-07 13:00:25.491347892 +0000 UTC m=+47.171733641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wj79l" (UID: "852d1deb-e626-454d-a9b0-19759119b16d") : secret "insights-runtime-extractor-tls" not found Sep 07 13:00:24.738112 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:24.738031 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vswfs_2fddfa4d-1664-4139-8891-e7c6bc67fde4/dns-node-resolver/0.log" Sep 07 13:00:25.338548 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:25.338518 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hjp2h_9e73d458-8cb5-4913-9ad2-54f0615ef55d/node-ca/0.log" Sep 07 13:00:25.504918 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:25.504881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:25.504918 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:25.504929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:25.505124 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:25.505014 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 13:00:25.505124 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:25.505035 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 13:00:25.505124 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:25.505095 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls podName:852d1deb-e626-454d-a9b0-19759119b16d nodeName:}" failed. No retries permitted until 2026-09-07 13:00:29.505072966 +0000 UTC m=+51.185458715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wj79l" (UID: "852d1deb-e626-454d-a9b0-19759119b16d") : secret "insights-runtime-extractor-tls" not found Sep 07 13:00:25.505124 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:25.505115 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm podName:852d1deb-e626-454d-a9b0-19759119b16d nodeName:}" failed. No retries permitted until 2026-09-07 13:00:29.505103656 +0000 UTC m=+51.185489410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wj79l" (UID: "852d1deb-e626-454d-a9b0-19759119b16d") : configmap "kube-rbac-proxy" not found Sep 07 13:00:26.137265 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:26.137231 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2p52x_d48d2f85-2ba1-4b88-9f3d-f512294a07f3/serve-healthcheck-canary/0.log" Sep 07 13:00:26.739942 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:26.739913 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-xd8sb_a53dca13-f533-42fe-b3ba-99671f656e9d/migrator/0.log" Sep 07 13:00:26.938081 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:26.938055 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-xd8sb_a53dca13-f533-42fe-b3ba-99671f656e9d/graceful-termination/0.log" Sep 07 13:00:29.132374 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:29.132344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:29.134538 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:29.134518 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65861502-da2f-4c4d-a629-c860d4c3f6e9-metrics-tls\") pod \"dns-default-xg2sx\" (UID: \"65861502-da2f-4c4d-a629-c860d4c3f6e9\") " pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:29.146648 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:29.146628 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:29.450339 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:29.450295 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xg2sx"] Sep 07 13:00:29.455084 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:29.455061 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65861502_da2f_4c4d_a629_c860d4c3f6e9.slice/crio-abba31ff486d94178dfdf4c2cbc9fb0ace3e7ac08a862fc391faf4c4c2ee909c WatchSource:0}: Error finding container abba31ff486d94178dfdf4c2cbc9fb0ace3e7ac08a862fc391faf4c4c2ee909c: Status 404 returned error can't find the container with id abba31ff486d94178dfdf4c2cbc9fb0ace3e7ac08a862fc391faf4c4c2ee909c Sep 07 13:00:29.535539 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:29.535504 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:29.535700 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:29.535545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:29.535700 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:29.535644 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 07 13:00:29.535700 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:29.535667 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 07 13:00:29.535700 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:29.535702 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm podName:852d1deb-e626-454d-a9b0-19759119b16d nodeName:}" failed. No retries permitted until 2026-09-07 13:00:37.535685346 +0000 UTC m=+59.216071094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wj79l" (UID: "852d1deb-e626-454d-a9b0-19759119b16d") : configmap "kube-rbac-proxy" not found Sep 07 13:00:29.535862 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:29.535804 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls podName:852d1deb-e626-454d-a9b0-19759119b16d nodeName:}" failed. No retries permitted until 2026-09-07 13:00:37.535783648 +0000 UTC m=+59.216169403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wj79l" (UID: "852d1deb-e626-454d-a9b0-19759119b16d") : secret "insights-runtime-extractor-tls" not found Sep 07 13:00:29.738423 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:29.738339 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:29.738563 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:29.738430 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:29.738563 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:29.738506 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls podName:1ab8a00e-df57-45f1-b68c-2198e83caf41 nodeName:}" failed. No retries permitted until 2026-09-07 13:00:45.738489814 +0000 UTC m=+67.418875565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q8dpn" (UID: "1ab8a00e-df57-45f1-b68c-2198e83caf41") : secret "cluster-monitoring-operator-tls" not found Sep 07 13:00:29.940310 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:29.940273 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:29.940450 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:29.940334 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:29.940450 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:29.940422 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 07 13:00:29.940540 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:00:29.940490 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert podName:974b5145-6856-4fea-b95d-e393bd76baed nodeName:}" failed. No retries permitted until 2026-09-07 13:00:45.94047272 +0000 UTC m=+67.620858468 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-kwlqf" (UID: "974b5145-6856-4fea-b95d-e393bd76baed") : secret "networking-console-plugin-cert" not found Sep 07 13:00:29.942785 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:29.942766 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls\") pod \"image-registry-696c9f54dc-grjtd\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:30.022406 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:30.022328 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:30.060929 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:30.060890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xg2sx" event={"ID":"65861502-da2f-4c4d-a629-c860d4c3f6e9","Type":"ContainerStarted","Data":"abba31ff486d94178dfdf4c2cbc9fb0ace3e7ac08a862fc391faf4c4c2ee909c"} Sep 07 13:00:30.299387 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:30.299301 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-696c9f54dc-grjtd"] Sep 07 13:00:30.302027 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:30.302001 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4212367c_1066_4682_859a_0abda62d959a.slice/crio-7dc69deed6561aba0abe85972c700db0c20ec4dec7d5e099faafd0a6c2cc49e8 WatchSource:0}: Error finding container 7dc69deed6561aba0abe85972c700db0c20ec4dec7d5e099faafd0a6c2cc49e8: Status 404 returned error can't find the container with id 7dc69deed6561aba0abe85972c700db0c20ec4dec7d5e099faafd0a6c2cc49e8 Sep 07 13:00:31.064742 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:31.064707 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" event={"ID":"4212367c-1066-4682-859a-0abda62d959a","Type":"ContainerStarted","Data":"4ec08cda8f146f192716998d71b5e91ec3e8c62bce399cd72a550d2621bde445"} Sep 07 13:00:31.064933 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:31.064757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" event={"ID":"4212367c-1066-4682-859a-0abda62d959a","Type":"ContainerStarted","Data":"7dc69deed6561aba0abe85972c700db0c20ec4dec7d5e099faafd0a6c2cc49e8"} Sep 07 13:00:31.064933 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:31.064827 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:31.094659 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:31.094612 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" podStartSLOduration=17.094577971 podStartE2EDuration="17.094577971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:31.093693648 +0000 UTC m=+52.774079418" watchObservedRunningTime="2026-09-07 13:00:31.094577971 +0000 UTC m=+52.774963743" Sep 07 13:00:33.073987 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:33.073937 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xg2sx" event={"ID":"65861502-da2f-4c4d-a629-c860d4c3f6e9","Type":"ContainerStarted","Data":"582a5e6b6655c42c2264a839de4b2a94655e79e8563f9e74b55c9f132c109836"} Sep 07 13:00:33.074528 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:33.073999 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xg2sx" event={"ID":"65861502-da2f-4c4d-a629-c860d4c3f6e9","Type":"ContainerStarted","Data":"e6d7b8a718eb686eb6d2b74dc4f9e9bf6e52cc4841829642e69d1e393c6a6757"} Sep 07 13:00:33.074528 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:33.074079 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:33.106988 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:33.106939 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xg2sx" podStartSLOduration=17.276003538 podStartE2EDuration="20.106925843s" totalImagesPullingTime="2.830922305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:33.105567932 +0000 UTC m=+54.785953701" watchObservedRunningTime="2026-09-07 13:00:33.106925843 +0000 UTC m=+54.787311613" Sep 07 13:00:37.600714 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:37.600681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:37.601062 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:37.600728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:37.601207 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:37.601187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852d1deb-e626-454d-a9b0-19759119b16d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:37.602922 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:37.602897 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852d1deb-e626-454d-a9b0-19759119b16d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wj79l\" (UID: \"852d1deb-e626-454d-a9b0-19759119b16d\") " pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:37.707668 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:37.707617 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wj79l" Sep 07 13:00:37.976953 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:37.976920 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wj79l"] Sep 07 13:00:37.979989 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:37.979960 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852d1deb_e626_454d_a9b0_19759119b16d.slice/crio-ece523e06feb78c2644b53117bd15d861142a665e8618b829a488b74b784867f WatchSource:0}: Error finding container ece523e06feb78c2644b53117bd15d861142a665e8618b829a488b74b784867f: Status 404 returned error can't find the container with id ece523e06feb78c2644b53117bd15d861142a665e8618b829a488b74b784867f Sep 07 13:00:37.995317 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:37.995295 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qs8fs" Sep 07 13:00:38.094613 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:38.094565 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wj79l" event={"ID":"852d1deb-e626-454d-a9b0-19759119b16d","Type":"ContainerStarted","Data":"a59765245c96bed7322087886edfb86fdd431a7998f13867a3b1bc4d90b25821"} Sep 07 13:00:38.094745 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:38.094638 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wj79l" event={"ID":"852d1deb-e626-454d-a9b0-19759119b16d","Type":"ContainerStarted","Data":"ece523e06feb78c2644b53117bd15d861142a665e8618b829a488b74b784867f"} Sep 07 13:00:40.103762 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:40.103719 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wj79l" event={"ID":"852d1deb-e626-454d-a9b0-19759119b16d","Type":"ContainerStarted","Data":"f293f3df4ccdcc5f351c119978ae4dc3921430dad1087e5c2cebfeb24fde2638"} Sep 07 13:00:41.108620 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:41.108523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wj79l" event={"ID":"852d1deb-e626-454d-a9b0-19759119b16d","Type":"ContainerStarted","Data":"d4fbb4ecba94c1d6f21e63fd282c43c21f4d925fdeeefe70a7a23b90e69d8d33"} Sep 07 13:00:41.139420 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:41.139372 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wj79l" podStartSLOduration=17.396337116 podStartE2EDuration="20.139358544s" totalImagesPullingTime="2.743021428s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 13:00:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:41.137789524 +0000 UTC m=+62.818175294" watchObservedRunningTime="2026-09-07 13:00:41.139358544 +0000 UTC m=+62.819744313" Sep 07 13:00:43.079189 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:43.079154 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xg2sx" Sep 07 13:00:44.552073 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.552031 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:44.554667 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.554649 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 13:00:44.565029 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.565000 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d7435f-c609-4aea-b385-72e0d14076ba-metrics-certs\") pod \"network-metrics-daemon-6wqht\" (UID: \"62d7435f-c609-4aea-b385-72e0d14076ba\") " pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:44.652516 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.652484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:44.654961 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.654944 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 13:00:44.665078 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.665053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2143bf9-8156-489e-9d51-cdee196d4ea7-original-pull-secret\") pod \"global-pull-secret-syncer-7nlgb\" (UID: \"b2143bf9-8156-489e-9d51-cdee196d4ea7\") " pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:44.732384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.732354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7nlgb" Sep 07 13:00:44.753198 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.753168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgsj\" (UniqueName: \"kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj\") pod \"network-check-target-tq7v6\" (UID: \"783539e3-6dae-4567-ba41-f9627dddc085\") " pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:44.755668 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.755646 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 07 13:00:44.765840 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.765811 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 07 13:00:44.777050 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.777025 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krgsj\" (UniqueName: \"kubernetes.io/projected/783539e3-6dae-4567-ba41-f9627dddc085-kube-api-access-krgsj\") pod \"network-check-target-tq7v6\" (UID: \"783539e3-6dae-4567-ba41-f9627dddc085\") " pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:44.836804 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.836732 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5v9xw\"" Sep 07 13:00:44.844671 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:44.844654 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wqht" Sep 07 13:00:45.000487 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:45.000460 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7nlgb"] Sep 07 13:00:45.002859 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:45.002833 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2143bf9_8156_489e_9d51_cdee196d4ea7.slice/crio-39a568d003ade70729280d4c4f5ce516006b13e69c48cb0a2e1673d5dd6ce0c5 WatchSource:0}: Error finding container 39a568d003ade70729280d4c4f5ce516006b13e69c48cb0a2e1673d5dd6ce0c5: Status 404 returned error can't find the container with id 39a568d003ade70729280d4c4f5ce516006b13e69c48cb0a2e1673d5dd6ce0c5 Sep 07 13:00:45.041435 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:45.041409 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8d5qt\"" Sep 07 13:00:45.049415 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:45.049400 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:45.111106 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:45.111081 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6wqht"] Sep 07 13:00:45.115196 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:45.115170 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d7435f_c609_4aea_b385_72e0d14076ba.slice/crio-28b102fa1c3a1a32e27b3cac1319c03ad2c9774a21f15529869c22ef756e539b WatchSource:0}: Error finding container 28b102fa1c3a1a32e27b3cac1319c03ad2c9774a21f15529869c22ef756e539b: Status 404 returned error can't find the container with id 28b102fa1c3a1a32e27b3cac1319c03ad2c9774a21f15529869c22ef756e539b Sep 07 13:00:45.120478 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:45.120449 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wqht" event={"ID":"62d7435f-c609-4aea-b385-72e0d14076ba","Type":"ContainerStarted","Data":"28b102fa1c3a1a32e27b3cac1319c03ad2c9774a21f15529869c22ef756e539b"} Sep 07 13:00:45.121530 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:45.121509 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7nlgb" event={"ID":"b2143bf9-8156-489e-9d51-cdee196d4ea7","Type":"ContainerStarted","Data":"39a568d003ade70729280d4c4f5ce516006b13e69c48cb0a2e1673d5dd6ce0c5"} Sep 07 13:00:45.315477 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:45.315447 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tq7v6"] Sep 07 13:00:45.318582 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:45.318551 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783539e3_6dae_4567_ba41_f9627dddc085.slice/crio-bed66424ed8ee5479788978e6c47cc691d8586ba6d4500b0e2b26b504aff11e0 WatchSource:0}: Error finding container bed66424ed8ee5479788978e6c47cc691d8586ba6d4500b0e2b26b504aff11e0: Status 404 returned error can't find the container with id bed66424ed8ee5479788978e6c47cc691d8586ba6d4500b0e2b26b504aff11e0 Sep 07 13:00:45.760694 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:45.760652 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:45.763655 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:45.763629 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ab8a00e-df57-45f1-b68c-2198e83caf41-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q8dpn\" (UID: \"1ab8a00e-df57-45f1-b68c-2198e83caf41\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:45.961681 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:45.961637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:45.964683 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:45.964650 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/974b5145-6856-4fea-b95d-e393bd76baed-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-kwlqf\" (UID: \"974b5145-6856-4fea-b95d-e393bd76baed\") " pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:46.017719 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.017637 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pp6wd\"" Sep 07 13:00:46.027102 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.026544 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" Sep 07 13:00:46.035075 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.035046 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl"] Sep 07 13:00:46.038446 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.038425 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:00:46.042213 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.042179 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 07 13:00:46.042321 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.042191 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 07 13:00:46.042321 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.042193 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 07 13:00:46.042979 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.042960 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 07 13:00:46.045939 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.045918 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-696c9f54dc-grjtd"] Sep 07 13:00:46.053754 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.053650 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl"] Sep 07 13:00:46.055331 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.055208 2565 patch_prober.go:28] interesting pod/image-registry-696c9f54dc-grjtd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 07 13:00:46.055331 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.055267 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" podUID="4212367c-1066-4682-859a-0abda62d959a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 13:00:46.127054 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.127018 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tq7v6" event={"ID":"783539e3-6dae-4567-ba41-f9627dddc085","Type":"ContainerStarted","Data":"bed66424ed8ee5479788978e6c47cc691d8586ba6d4500b0e2b26b504aff11e0"} Sep 07 13:00:46.130607 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.130572 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kcxq5\"" Sep 07 13:00:46.138465 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.138444 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" Sep 07 13:00:46.163988 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.163935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39fa15c7-359f-420b-aeb2-f97ab80eb386-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddcfd857c-8gghl\" (UID: \"39fa15c7-359f-420b-aeb2-f97ab80eb386\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:00:46.163988 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.163987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39fa15c7-359f-420b-aeb2-f97ab80eb386-tmp\") pod \"klusterlet-addon-workmgr-7ddcfd857c-8gghl\" (UID: \"39fa15c7-359f-420b-aeb2-f97ab80eb386\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:00:46.164254 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.164030 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lqw4\" (UniqueName: \"kubernetes.io/projected/39fa15c7-359f-420b-aeb2-f97ab80eb386-kube-api-access-9lqw4\") pod \"klusterlet-addon-workmgr-7ddcfd857c-8gghl\" (UID: \"39fa15c7-359f-420b-aeb2-f97ab80eb386\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:00:46.164950 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.164923 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65cccd88c9-q57vf"] Sep 07 13:00:46.168273 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.168047 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.173070 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.172357 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-s42nz"] Sep 07 13:00:46.174207 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.174188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-s42nz" Sep 07 13:00:46.176872 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.176851 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 07 13:00:46.176872 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.176869 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lmrpz\"" Sep 07 13:00:46.178464 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.178440 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 07 13:00:46.195988 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.195929 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65cccd88c9-q57vf"] Sep 07 13:00:46.200345 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.200247 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-s42nz"] Sep 07 13:00:46.265108 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.264891 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34d5e8f1-56b0-4169-86e0-7418b9093a50-registry-certificates\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.265108 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.264938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39fa15c7-359f-420b-aeb2-f97ab80eb386-tmp\") pod \"klusterlet-addon-workmgr-7ddcfd857c-8gghl\" (UID: \"39fa15c7-359f-420b-aeb2-f97ab80eb386\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:00:46.265108 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.264970 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8b4\" (UniqueName: \"kubernetes.io/projected/1edf6720-0ac6-4149-8bee-e617b17c9613-kube-api-access-8d8b4\") pod \"downloads-85d98c998f-s42nz\" (UID: \"1edf6720-0ac6-4149-8bee-e617b17c9613\") " pod="openshift-console/downloads-85d98c998f-s42nz" Sep 07 13:00:46.265108 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.264998 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34d5e8f1-56b0-4169-86e0-7418b9093a50-bound-sa-token\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.265428 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.265130 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34d5e8f1-56b0-4169-86e0-7418b9093a50-installation-pull-secrets\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.265428 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.265162 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ncvb\" (UniqueName: \"kubernetes.io/projected/34d5e8f1-56b0-4169-86e0-7418b9093a50-kube-api-access-8ncvb\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.265428 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.265186 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39fa15c7-359f-420b-aeb2-f97ab80eb386-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddcfd857c-8gghl\" (UID: \"39fa15c7-359f-420b-aeb2-f97ab80eb386\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:00:46.265428 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.265228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lqw4\" (UniqueName: \"kubernetes.io/projected/39fa15c7-359f-420b-aeb2-f97ab80eb386-kube-api-access-9lqw4\") pod \"klusterlet-addon-workmgr-7ddcfd857c-8gghl\" (UID: \"39fa15c7-359f-420b-aeb2-f97ab80eb386\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:00:46.265428 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.265262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34d5e8f1-56b0-4169-86e0-7418b9093a50-image-registry-private-configuration\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.265428 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.265285 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e8f1-56b0-4169-86e0-7418b9093a50-trusted-ca\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.265428 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.265318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34d5e8f1-56b0-4169-86e0-7418b9093a50-registry-tls\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.265428 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.265340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34d5e8f1-56b0-4169-86e0-7418b9093a50-ca-trust-extracted\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.266158 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.266125 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39fa15c7-359f-420b-aeb2-f97ab80eb386-tmp\") pod \"klusterlet-addon-workmgr-7ddcfd857c-8gghl\" (UID: \"39fa15c7-359f-420b-aeb2-f97ab80eb386\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:00:46.268852 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.268780 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39fa15c7-359f-420b-aeb2-f97ab80eb386-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddcfd857c-8gghl\" (UID: \"39fa15c7-359f-420b-aeb2-f97ab80eb386\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:00:46.276768 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.276723 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lqw4\" (UniqueName: \"kubernetes.io/projected/39fa15c7-359f-420b-aeb2-f97ab80eb386-kube-api-access-9lqw4\") pod \"klusterlet-addon-workmgr-7ddcfd857c-8gghl\" (UID: \"39fa15c7-359f-420b-aeb2-f97ab80eb386\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:00:46.328791 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.328685 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn"] Sep 07 13:00:46.364510 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:46.364473 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab8a00e_df57_45f1_b68c_2198e83caf41.slice/crio-9b69b873b901efe91d99b5c324c47213671e206acd6351d3bcdc6e399231bba7 WatchSource:0}: Error finding container 9b69b873b901efe91d99b5c324c47213671e206acd6351d3bcdc6e399231bba7: Status 404 returned error can't find the container with id 9b69b873b901efe91d99b5c324c47213671e206acd6351d3bcdc6e399231bba7 Sep 07 13:00:46.366113 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.366063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ncvb\" (UniqueName: \"kubernetes.io/projected/34d5e8f1-56b0-4169-86e0-7418b9093a50-kube-api-access-8ncvb\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.366193 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.366109 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34d5e8f1-56b0-4169-86e0-7418b9093a50-image-registry-private-configuration\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.366193 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.366136 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e8f1-56b0-4169-86e0-7418b9093a50-trusted-ca\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.366193 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.366167 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34d5e8f1-56b0-4169-86e0-7418b9093a50-registry-tls\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.366408 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.366193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34d5e8f1-56b0-4169-86e0-7418b9093a50-ca-trust-extracted\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.366408 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.366226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34d5e8f1-56b0-4169-86e0-7418b9093a50-registry-certificates\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.366408 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.366255 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8b4\" (UniqueName: \"kubernetes.io/projected/1edf6720-0ac6-4149-8bee-e617b17c9613-kube-api-access-8d8b4\") pod \"downloads-85d98c998f-s42nz\" (UID: \"1edf6720-0ac6-4149-8bee-e617b17c9613\") " pod="openshift-console/downloads-85d98c998f-s42nz" Sep 07 13:00:46.366408 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.366283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34d5e8f1-56b0-4169-86e0-7418b9093a50-bound-sa-token\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.366408 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.366324 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34d5e8f1-56b0-4169-86e0-7418b9093a50-installation-pull-secrets\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.367785 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.367762 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34d5e8f1-56b0-4169-86e0-7418b9093a50-registry-certificates\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.367901 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.367874 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34d5e8f1-56b0-4169-86e0-7418b9093a50-ca-trust-extracted\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.368206 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.368133 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34d5e8f1-56b0-4169-86e0-7418b9093a50-trusted-ca\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.369877 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.369857 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:00:46.370103 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.370086 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34d5e8f1-56b0-4169-86e0-7418b9093a50-registry-tls\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.370270 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.370192 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34d5e8f1-56b0-4169-86e0-7418b9093a50-image-registry-private-configuration\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.371561 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.371501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34d5e8f1-56b0-4169-86e0-7418b9093a50-installation-pull-secrets\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.383167 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.383122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8b4\" (UniqueName: \"kubernetes.io/projected/1edf6720-0ac6-4149-8bee-e617b17c9613-kube-api-access-8d8b4\") pod \"downloads-85d98c998f-s42nz\" (UID: \"1edf6720-0ac6-4149-8bee-e617b17c9613\") " pod="openshift-console/downloads-85d98c998f-s42nz" Sep 07 13:00:46.386525 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.386498 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34d5e8f1-56b0-4169-86e0-7418b9093a50-bound-sa-token\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.386852 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.386796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ncvb\" (UniqueName: \"kubernetes.io/projected/34d5e8f1-56b0-4169-86e0-7418b9093a50-kube-api-access-8ncvb\") pod \"image-registry-65cccd88c9-q57vf\" (UID: \"34d5e8f1-56b0-4169-86e0-7418b9093a50\") " pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.428976 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.428891 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-kwlqf"] Sep 07 13:00:46.437823 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:46.437366 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974b5145_6856_4fea_b95d_e393bd76baed.slice/crio-521826ecbeb1ed0e577baa3d7bdb6c4a09dbcdf8a67727cd7abf4156a768b4de WatchSource:0}: Error finding container 521826ecbeb1ed0e577baa3d7bdb6c4a09dbcdf8a67727cd7abf4156a768b4de: Status 404 returned error can't find the container with id 521826ecbeb1ed0e577baa3d7bdb6c4a09dbcdf8a67727cd7abf4156a768b4de Sep 07 13:00:46.483320 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.483289 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:46.490212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.490187 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-s42nz" Sep 07 13:00:46.659722 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.659559 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl"] Sep 07 13:00:46.663812 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:46.663780 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fa15c7_359f_420b_aeb2_f97ab80eb386.slice/crio-a571faef8c3ad3055c2b1b5b2d7159831209a2134d76af393e3e9c69f88730d5 WatchSource:0}: Error finding container a571faef8c3ad3055c2b1b5b2d7159831209a2134d76af393e3e9c69f88730d5: Status 404 returned error can't find the container with id a571faef8c3ad3055c2b1b5b2d7159831209a2134d76af393e3e9c69f88730d5 Sep 07 13:00:46.773526 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.773452 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65cccd88c9-q57vf"] Sep 07 13:00:46.796262 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:46.796146 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-s42nz"] Sep 07 13:00:46.993132 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:46.992848 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d5e8f1_56b0_4169_86e0_7418b9093a50.slice/crio-2506e3e2fa441e0e72371beeec48e323928de127ae88c611eb01ca0d7a44fbd4 WatchSource:0}: Error finding container 2506e3e2fa441e0e72371beeec48e323928de127ae88c611eb01ca0d7a44fbd4: Status 404 returned error can't find the container with id 2506e3e2fa441e0e72371beeec48e323928de127ae88c611eb01ca0d7a44fbd4 Sep 07 13:00:46.993372 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:00:46.993326 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edf6720_0ac6_4149_8bee_e617b17c9613.slice/crio-9067908a3336dc1c16020f8d778d6af8155a27d66175aed776aa50f2812a9b4c WatchSource:0}: Error finding container 9067908a3336dc1c16020f8d778d6af8155a27d66175aed776aa50f2812a9b4c: Status 404 returned error can't find the container with id 9067908a3336dc1c16020f8d778d6af8155a27d66175aed776aa50f2812a9b4c Sep 07 13:00:47.136283 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:47.136219 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" event={"ID":"34d5e8f1-56b0-4169-86e0-7418b9093a50","Type":"ContainerStarted","Data":"f4f279a5c9507deb16c758fb21733fb8b127d10d2a427d37cad4e73e6c00a79c"} Sep 07 13:00:47.136459 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:47.136334 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" event={"ID":"34d5e8f1-56b0-4169-86e0-7418b9093a50","Type":"ContainerStarted","Data":"2506e3e2fa441e0e72371beeec48e323928de127ae88c611eb01ca0d7a44fbd4"} Sep 07 13:00:47.136552 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:47.136534 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:00:47.138911 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:47.138831 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-s42nz" event={"ID":"1edf6720-0ac6-4149-8bee-e617b17c9613","Type":"ContainerStarted","Data":"9067908a3336dc1c16020f8d778d6af8155a27d66175aed776aa50f2812a9b4c"} Sep 07 13:00:47.140796 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:47.140765 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" event={"ID":"39fa15c7-359f-420b-aeb2-f97ab80eb386","Type":"ContainerStarted","Data":"a571faef8c3ad3055c2b1b5b2d7159831209a2134d76af393e3e9c69f88730d5"} Sep 07 13:00:47.142641 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:47.142610 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" event={"ID":"974b5145-6856-4fea-b95d-e393bd76baed","Type":"ContainerStarted","Data":"521826ecbeb1ed0e577baa3d7bdb6c4a09dbcdf8a67727cd7abf4156a768b4de"} Sep 07 13:00:47.144428 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:47.144402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" event={"ID":"1ab8a00e-df57-45f1-b68c-2198e83caf41","Type":"ContainerStarted","Data":"9b69b873b901efe91d99b5c324c47213671e206acd6351d3bcdc6e399231bba7"} Sep 07 13:00:47.163463 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:47.163406 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" podStartSLOduration=1.16338693 podStartE2EDuration="1.16338693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:47.162329053 +0000 UTC m=+68.842714824" watchObservedRunningTime="2026-09-07 13:00:47.16338693 +0000 UTC m=+68.843772703" Sep 07 13:00:48.155766 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:48.155651 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wqht" event={"ID":"62d7435f-c609-4aea-b385-72e0d14076ba","Type":"ContainerStarted","Data":"54103a8f7dec66920df1a4708a4bd04c7c34848d30394fb679e9d93e4b6ceec3"} Sep 07 13:00:48.155766 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:48.155713 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wqht" event={"ID":"62d7435f-c609-4aea-b385-72e0d14076ba","Type":"ContainerStarted","Data":"4c6bb5e3ecd813c3df7f3b8e773a7cdb9cc00451e79329f6d5dad551ff93f0f5"} Sep 07 13:00:48.173078 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:48.172276 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6wqht" podStartSLOduration=68.222624917 podStartE2EDuration="1m10.172253566s" totalImagesPullingTime="1.949628649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:48.171524814 +0000 UTC m=+69.851910586" watchObservedRunningTime="2026-09-07 13:00:48.172253566 +0000 UTC m=+69.852639337" Sep 07 13:00:52.173538 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:52.173467 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" event={"ID":"974b5145-6856-4fea-b95d-e393bd76baed","Type":"ContainerStarted","Data":"54a3962b114f307f506e17cd5375578e4cb3a5e6f7ac4b7863c4d79ba606f20d"} Sep 07 13:00:52.175441 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:52.175415 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" event={"ID":"1ab8a00e-df57-45f1-b68c-2198e83caf41","Type":"ContainerStarted","Data":"a74a2b016d8e2f979e4b22dfa17caa28c96685f80e324c407a2b9ee9fd845f6d"} Sep 07 13:00:52.177347 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:52.177304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7nlgb" event={"ID":"b2143bf9-8156-489e-9d51-cdee196d4ea7","Type":"ContainerStarted","Data":"55bd62f51dedba0d11f42b12bef81de822b010bcbdd7151f3464b0d07306e5b8"} Sep 07 13:00:52.178991 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:52.178967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tq7v6" event={"ID":"783539e3-6dae-4567-ba41-f9627dddc085","Type":"ContainerStarted","Data":"82171deb9126f286c4d62c469c77dfe49e4059b59952d0ed5982ecc12e03a5d6"} Sep 07 13:00:52.179122 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:52.179104 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:00:52.196558 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:52.196505 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-kwlqf" podStartSLOduration=33.737130186 podStartE2EDuration="39.196485887s" totalImagesPullingTime="5.459355701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:52.19491942 +0000 UTC m=+73.875305191" watchObservedRunningTime="2026-09-07 13:00:52.196485887 +0000 UTC m=+73.876871658" Sep 07 13:00:52.225571 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:52.225248 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q8dpn" podStartSLOduration=33.687354918 podStartE2EDuration="39.225224443s" totalImagesPullingTime="5.537869525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:52.221642556 +0000 UTC m=+73.902028327" watchObservedRunningTime="2026-09-07 13:00:52.225224443 +0000 UTC m=+73.905610214" Sep 07 13:00:52.247848 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:52.247792 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7nlgb" podStartSLOduration=66.332260845 podStartE2EDuration="1m13.247772551s" totalImagesPullingTime="6.915511706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 12:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:52.2466883 +0000 UTC m=+73.927074071" watchObservedRunningTime="2026-09-07 13:00:52.247772551 +0000 UTC m=+73.928158325" Sep 07 13:00:52.269093 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:52.269034 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tq7v6" podStartSLOduration=66.678255335 podStartE2EDuration="1m13.269010796s" totalImagesPullingTime="6.590755461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 12:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:00:52.267863236 +0000 UTC m=+73.948249007" watchObservedRunningTime="2026-09-07 13:00:52.269010796 +0000 UTC m=+73.949396567" Sep 07 13:00:56.051317 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.051286 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:00:56.599968 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.599926 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-n65rb"] Sep 07 13:00:56.602506 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.602473 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.605776 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.605739 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 07 13:00:56.605896 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.605799 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gp6md\"" Sep 07 13:00:56.605896 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.605739 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 07 13:00:56.606065 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.606049 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 07 13:00:56.617288 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.617262 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-n65rb"] Sep 07 13:00:56.756794 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.756755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfbbf363-2a82-4f30-88a0-c28dff487e42-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.756794 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.756805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr97p\" (UniqueName: \"kubernetes.io/projected/bfbbf363-2a82-4f30-88a0-c28dff487e42-kube-api-access-vr97p\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.757046 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.756847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfbbf363-2a82-4f30-88a0-c28dff487e42-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.757046 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.756886 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfbbf363-2a82-4f30-88a0-c28dff487e42-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.857978 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.857884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr97p\" (UniqueName: \"kubernetes.io/projected/bfbbf363-2a82-4f30-88a0-c28dff487e42-kube-api-access-vr97p\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.857978 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.857932 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfbbf363-2a82-4f30-88a0-c28dff487e42-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.858177 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.858115 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfbbf363-2a82-4f30-88a0-c28dff487e42-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.858235 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.858221 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfbbf363-2a82-4f30-88a0-c28dff487e42-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.858908 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.858878 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfbbf363-2a82-4f30-88a0-c28dff487e42-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.860632 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.860610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfbbf363-2a82-4f30-88a0-c28dff487e42-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.860797 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.860780 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bfbbf363-2a82-4f30-88a0-c28dff487e42-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.871110 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.871085 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr97p\" (UniqueName: \"kubernetes.io/projected/bfbbf363-2a82-4f30-88a0-c28dff487e42-kube-api-access-vr97p\") pod \"prometheus-operator-6f4946db56-n65rb\" (UID: \"bfbbf363-2a82-4f30-88a0-c28dff487e42\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:00:56.915124 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:00:56.915083 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" Sep 07 13:01:06.421885 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:06.421811 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-n65rb"] Sep 07 13:01:06.480255 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:01:06.480219 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbbf363_2a82_4f30_88a0_c28dff487e42.slice/crio-6f3443193f667e05df863bab4abcab113684b5bb10a3b1b3487a9a4a3376830e WatchSource:0}: Error finding container 6f3443193f667e05df863bab4abcab113684b5bb10a3b1b3487a9a4a3376830e: Status 404 returned error can't find the container with id 6f3443193f667e05df863bab4abcab113684b5bb10a3b1b3487a9a4a3376830e Sep 07 13:01:07.243051 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:07.243012 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-s42nz" event={"ID":"1edf6720-0ac6-4149-8bee-e617b17c9613","Type":"ContainerStarted","Data":"f3befa01b0c98425d137a119a32b7eef497a694f87bd2100fea1ae56f1f6f75c"} Sep 07 13:01:07.243327 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:07.243307 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-s42nz" Sep 07 13:01:07.245329 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:07.245295 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" event={"ID":"39fa15c7-359f-420b-aeb2-f97ab80eb386","Type":"ContainerStarted","Data":"f8a9d62e873a313bfff867e7b48cf9b0722a7aec7e031080dad57982b644e5c7"} Sep 07 13:01:07.245790 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:07.245769 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:01:07.246999 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:07.246971 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" event={"ID":"bfbbf363-2a82-4f30-88a0-c28dff487e42","Type":"ContainerStarted","Data":"6f3443193f667e05df863bab4abcab113684b5bb10a3b1b3487a9a4a3376830e"} Sep 07 13:01:07.247214 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:07.247196 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" Sep 07 13:01:07.256271 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:07.256244 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-s42nz" Sep 07 13:01:07.263395 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:07.263342 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-s42nz" podStartSLOduration=2.058205454 podStartE2EDuration="21.26332719s" totalImagesPullingTime="19.205121736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:01:07.262784381 +0000 UTC m=+88.943170154" watchObservedRunningTime="2026-09-07 13:01:07.26332719 +0000 UTC m=+88.943712960" Sep 07 13:01:08.160765 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:08.160735 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65cccd88c9-q57vf" Sep 07 13:01:08.183975 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:08.183922 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddcfd857c-8gghl" podStartSLOduration=2.343055423 podStartE2EDuration="22.183902675s" totalImagesPullingTime="19.840847252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:01:07.301937059 +0000 UTC m=+88.982322829" watchObservedRunningTime="2026-09-07 13:01:08.183902675 +0000 UTC m=+89.864288445" Sep 07 13:01:10.265572 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:10.265538 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" event={"ID":"bfbbf363-2a82-4f30-88a0-c28dff487e42","Type":"ContainerStarted","Data":"1990a770d369b7cf7c72ecc2644227093999f3a855db837a95b4e0e674fa12d5"} Sep 07 13:01:11.082631 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.082566 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" podUID="4212367c-1066-4682-859a-0abda62d959a" containerName="registry" containerID="cri-o://4ec08cda8f146f192716998d71b5e91ec3e8c62bce399cd72a550d2621bde445" gracePeriod=30 Sep 07 13:01:11.272677 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.272127 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" event={"ID":"bfbbf363-2a82-4f30-88a0-c28dff487e42","Type":"ContainerStarted","Data":"6cdcc516136b0fa4f4471d63b57f1061056d3557722d11cebb7af58f344d277f"} Sep 07 13:01:11.273984 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.273959 2565 generic.go:356] "Generic (PLEG): container finished" podID="4212367c-1066-4682-859a-0abda62d959a" containerID="4ec08cda8f146f192716998d71b5e91ec3e8c62bce399cd72a550d2621bde445" exitCode=0 Sep 07 13:01:11.274106 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.274040 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" event={"ID":"4212367c-1066-4682-859a-0abda62d959a","Type":"ContainerDied","Data":"4ec08cda8f146f192716998d71b5e91ec3e8c62bce399cd72a550d2621bde445"} Sep 07 13:01:11.293888 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.293198 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-n65rb" podStartSLOduration=11.739728286 podStartE2EDuration="15.293174234s" totalImagesPullingTime="3.553445948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:01:11.29225577 +0000 UTC m=+92.972641543" watchObservedRunningTime="2026-09-07 13:01:11.293174234 +0000 UTC m=+92.973560005" Sep 07 13:01:11.374828 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.374798 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:01:11.477731 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.477692 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4212367c-1066-4682-859a-0abda62d959a-ca-trust-extracted\") pod \"4212367c-1066-4682-859a-0abda62d959a\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " Sep 07 13:01:11.477921 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.477748 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-image-registry-private-configuration\") pod \"4212367c-1066-4682-859a-0abda62d959a\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " Sep 07 13:01:11.477921 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.477786 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-trusted-ca\") pod \"4212367c-1066-4682-859a-0abda62d959a\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " Sep 07 13:01:11.477921 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.477814 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-registry-certificates\") pod \"4212367c-1066-4682-859a-0abda62d959a\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " Sep 07 13:01:11.477921 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.477863 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqrj6\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-kube-api-access-dqrj6\") pod \"4212367c-1066-4682-859a-0abda62d959a\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " Sep 07 13:01:11.477921 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.477888 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls\") pod \"4212367c-1066-4682-859a-0abda62d959a\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " Sep 07 13:01:11.477921 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.477908 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-bound-sa-token\") pod \"4212367c-1066-4682-859a-0abda62d959a\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " Sep 07 13:01:11.478232 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.477953 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-installation-pull-secrets\") pod \"4212367c-1066-4682-859a-0abda62d959a\" (UID: \"4212367c-1066-4682-859a-0abda62d959a\") " Sep 07 13:01:11.478286 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.478223 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4212367c-1066-4682-859a-0abda62d959a" (UID: "4212367c-1066-4682-859a-0abda62d959a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:01:11.478578 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.478541 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4212367c-1066-4682-859a-0abda62d959a" (UID: "4212367c-1066-4682-859a-0abda62d959a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:01:11.481463 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.481359 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4212367c-1066-4682-859a-0abda62d959a" (UID: "4212367c-1066-4682-859a-0abda62d959a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:01:11.481463 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.481397 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-kube-api-access-dqrj6" (OuterVolumeSpecName: "kube-api-access-dqrj6") pod "4212367c-1066-4682-859a-0abda62d959a" (UID: "4212367c-1066-4682-859a-0abda62d959a"). InnerVolumeSpecName "kube-api-access-dqrj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:01:11.481667 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.481546 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4212367c-1066-4682-859a-0abda62d959a" (UID: "4212367c-1066-4682-859a-0abda62d959a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:01:11.481667 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.481580 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4212367c-1066-4682-859a-0abda62d959a" (UID: "4212367c-1066-4682-859a-0abda62d959a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:01:11.481876 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.481852 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4212367c-1066-4682-859a-0abda62d959a" (UID: "4212367c-1066-4682-859a-0abda62d959a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:01:11.489025 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.488998 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4212367c-1066-4682-859a-0abda62d959a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4212367c-1066-4682-859a-0abda62d959a" (UID: "4212367c-1066-4682-859a-0abda62d959a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:01:11.579305 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.579264 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-trusted-ca\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:01:11.579305 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.579301 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4212367c-1066-4682-859a-0abda62d959a-registry-certificates\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:01:11.579305 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.579311 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqrj6\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-kube-api-access-dqrj6\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:01:11.579569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.579320 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-registry-tls\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:01:11.579569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.579329 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4212367c-1066-4682-859a-0abda62d959a-bound-sa-token\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:01:11.579569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.579337 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-installation-pull-secrets\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:01:11.579569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.579350 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4212367c-1066-4682-859a-0abda62d959a-ca-trust-extracted\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:01:11.579569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:11.579366 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4212367c-1066-4682-859a-0abda62d959a-image-registry-private-configuration\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:01:12.279810 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:12.279774 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" Sep 07 13:01:12.280278 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:12.279816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-696c9f54dc-grjtd" event={"ID":"4212367c-1066-4682-859a-0abda62d959a","Type":"ContainerDied","Data":"7dc69deed6561aba0abe85972c700db0c20ec4dec7d5e099faafd0a6c2cc49e8"} Sep 07 13:01:12.280278 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:12.279881 2565 scope.go:117] "RemoveContainer" containerID="4ec08cda8f146f192716998d71b5e91ec3e8c62bce399cd72a550d2621bde445" Sep 07 13:01:12.304036 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:12.304005 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-696c9f54dc-grjtd"] Sep 07 13:01:12.310663 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:12.310636 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-696c9f54dc-grjtd"] Sep 07 13:01:12.826319 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:12.826285 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4212367c-1066-4682-859a-0abda62d959a" path="/var/lib/kubelet/pods/4212367c-1066-4682-859a-0abda62d959a/volumes" Sep 07 13:01:13.196982 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.196946 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-64j7z"] Sep 07 13:01:13.197294 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.197275 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4212367c-1066-4682-859a-0abda62d959a" containerName="registry" Sep 07 13:01:13.197294 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.197294 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4212367c-1066-4682-859a-0abda62d959a" containerName="registry" Sep 07 13:01:13.197445 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.197371 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4212367c-1066-4682-859a-0abda62d959a" containerName="registry" Sep 07 13:01:13.222578 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.222547 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.223695 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.223666 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7lsw4"] Sep 07 13:01:13.226720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.226696 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 07 13:01:13.226846 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.226775 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 07 13:01:13.227567 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.227546 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 07 13:01:13.227809 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.227788 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-l7ztq\"" Sep 07 13:01:13.246168 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.246141 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-64j7z"] Sep 07 13:01:13.246307 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.246274 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.248451 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.248430 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2jq42\"" Sep 07 13:01:13.249087 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.249061 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 07 13:01:13.249190 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.249179 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 07 13:01:13.249720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.249699 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 07 13:01:13.394486 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-tls\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394502 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh5vn\" (UniqueName: \"kubernetes.io/projected/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-api-access-bh5vn\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394579 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22a97db3-4fcb-4e13-b1cc-54ffd9845248-sys\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq7xn\" (UniqueName: \"kubernetes.io/projected/22a97db3-4fcb-4e13-b1cc-54ffd9845248-kube-api-access-sq7xn\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/22a97db3-4fcb-4e13-b1cc-54ffd9845248-root\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-accelerators-collector-config\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06b6ce26-951d-44da-ae1d-4b20971557fd-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-wtmp\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394866 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22a97db3-4fcb-4e13-b1cc-54ffd9845248-metrics-client-ca\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.394945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394890 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-textfile\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.395391 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.394938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.395391 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.395001 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/06b6ce26-951d-44da-ae1d-4b20971557fd-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.495816 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.495736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.495816 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.495773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/22a97db3-4fcb-4e13-b1cc-54ffd9845248-root\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.495816 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.495794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-accelerators-collector-config\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.495820 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.496107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.495850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06b6ce26-951d-44da-ae1d-4b20971557fd-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.496107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.495871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-wtmp\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.495895 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22a97db3-4fcb-4e13-b1cc-54ffd9845248-metrics-client-ca\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.495925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-textfile\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.495952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.496107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.495990 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/06b6ce26-951d-44da-ae1d-4b20971557fd-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.496107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496035 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-tls\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496064 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-wtmp\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh5vn\" (UniqueName: \"kubernetes.io/projected/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-api-access-bh5vn\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.496803 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22a97db3-4fcb-4e13-b1cc-54ffd9845248-sys\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496803 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496182 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq7xn\" (UniqueName: \"kubernetes.io/projected/22a97db3-4fcb-4e13-b1cc-54ffd9845248-kube-api-access-sq7xn\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496803 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496271 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-textfile\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496803 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496460 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-accelerators-collector-config\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496803 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496490 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/06b6ce26-951d-44da-ae1d-4b20971557fd-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.496803 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496559 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22a97db3-4fcb-4e13-b1cc-54ffd9845248-sys\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.496803 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496689 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/22a97db3-4fcb-4e13-b1cc-54ffd9845248-root\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.497151 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.496998 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06b6ce26-951d-44da-ae1d-4b20971557fd-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.497212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.497192 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.497581 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.497557 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22a97db3-4fcb-4e13-b1cc-54ffd9845248-metrics-client-ca\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.498863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.498835 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.499185 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.499146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-tls\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.499284 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.499225 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22a97db3-4fcb-4e13-b1cc-54ffd9845248-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.499373 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.499344 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.508002 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.507976 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq7xn\" (UniqueName: \"kubernetes.io/projected/22a97db3-4fcb-4e13-b1cc-54ffd9845248-kube-api-access-sq7xn\") pod \"node-exporter-7lsw4\" (UID: \"22a97db3-4fcb-4e13-b1cc-54ffd9845248\") " pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.510175 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.510152 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh5vn\" (UniqueName: \"kubernetes.io/projected/06b6ce26-951d-44da-ae1d-4b20971557fd-kube-api-access-bh5vn\") pod \"kube-state-metrics-74cb57df7-64j7z\" (UID: \"06b6ce26-951d-44da-ae1d-4b20971557fd\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.533896 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.533869 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" Sep 07 13:01:13.556873 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.556841 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7lsw4" Sep 07 13:01:13.582999 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:01:13.582967 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a97db3_4fcb_4e13_b1cc_54ffd9845248.slice/crio-0a290c8f0b36bf00830fee20b174c4bddfccc1faf4ee547a9214073c37d0e920 WatchSource:0}: Error finding container 0a290c8f0b36bf00830fee20b174c4bddfccc1faf4ee547a9214073c37d0e920: Status 404 returned error can't find the container with id 0a290c8f0b36bf00830fee20b174c4bddfccc1faf4ee547a9214073c37d0e920 Sep 07 13:01:13.840921 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:13.840839 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-64j7z"] Sep 07 13:01:13.845176 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:01:13.845137 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b6ce26_951d_44da_ae1d_4b20971557fd.slice/crio-1334768f805e98f44b264d56b240b5070c9dc9c653dab15ea10618b09a6df318 WatchSource:0}: Error finding container 1334768f805e98f44b264d56b240b5070c9dc9c653dab15ea10618b09a6df318: Status 404 returned error can't find the container with id 1334768f805e98f44b264d56b240b5070c9dc9c653dab15ea10618b09a6df318 Sep 07 13:01:14.293728 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:14.293632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" event={"ID":"06b6ce26-951d-44da-ae1d-4b20971557fd","Type":"ContainerStarted","Data":"1334768f805e98f44b264d56b240b5070c9dc9c653dab15ea10618b09a6df318"} Sep 07 13:01:14.295616 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:14.295566 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7lsw4" event={"ID":"22a97db3-4fcb-4e13-b1cc-54ffd9845248","Type":"ContainerStarted","Data":"0a290c8f0b36bf00830fee20b174c4bddfccc1faf4ee547a9214073c37d0e920"} Sep 07 13:01:16.217051 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.216712 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f656656f-7r4lc"] Sep 07 13:01:16.236929 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.236900 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.245699 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.245674 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 07 13:01:16.246054 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.246035 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 07 13:01:16.246180 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.246033 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 07 13:01:16.246490 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.246465 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-oqjlac5mgcsa\"" Sep 07 13:01:16.246590 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.246558 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 07 13:01:16.246738 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.246718 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gncsz\"" Sep 07 13:01:16.256785 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.256757 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f656656f-7r4lc"] Sep 07 13:01:16.257332 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.257310 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 07 13:01:16.309924 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.309885 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" event={"ID":"06b6ce26-951d-44da-ae1d-4b20971557fd","Type":"ContainerStarted","Data":"615c1a0cad6ad0bb205fb13b1d6dfe2c0fca56fe04762b5c7d341127f0472cff"} Sep 07 13:01:16.310076 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.309951 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" event={"ID":"06b6ce26-951d-44da-ae1d-4b20971557fd","Type":"ContainerStarted","Data":"1533452047ec6eac0d2f685ceaacb5b0e568e0c5bbea478fe5f8c9d9838ca5fb"} Sep 07 13:01:16.422730 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.422700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-tls\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.422806 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.422765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-grpc-tls\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.422851 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.422820 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.422887 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.422850 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.422928 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.422908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.422968 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.422951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aee2d72e-4493-453b-9be4-5b60e710cd2c-metrics-client-ca\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.423017 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.422983 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c2vn\" (UniqueName: \"kubernetes.io/projected/aee2d72e-4493-453b-9be4-5b60e710cd2c-kube-api-access-5c2vn\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.423060 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.423021 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.523584 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.523552 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-tls\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.523759 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.523627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-grpc-tls\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.523759 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.523674 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.523759 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.523700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.523759 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.523724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.523956 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.523901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aee2d72e-4493-453b-9be4-5b60e710cd2c-metrics-client-ca\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.523956 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.523938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c2vn\" (UniqueName: \"kubernetes.io/projected/aee2d72e-4493-453b-9be4-5b60e710cd2c-kube-api-access-5c2vn\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.524062 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.523976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.524780 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.524723 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aee2d72e-4493-453b-9be4-5b60e710cd2c-metrics-client-ca\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.526670 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.526633 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.526848 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.526826 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-tls\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.526977 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.526957 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-grpc-tls\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.527161 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.527136 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.527468 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.527445 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.527550 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.527532 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aee2d72e-4493-453b-9be4-5b60e710cd2c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.532744 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.532724 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c2vn\" (UniqueName: \"kubernetes.io/projected/aee2d72e-4493-453b-9be4-5b60e710cd2c-kube-api-access-5c2vn\") pod \"thanos-querier-5f656656f-7r4lc\" (UID: \"aee2d72e-4493-453b-9be4-5b60e710cd2c\") " pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.548646 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.548627 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:16.841376 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:16.841329 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f656656f-7r4lc"] Sep 07 13:01:16.848623 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:01:16.848582 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee2d72e_4493_453b_9be4_5b60e710cd2c.slice/crio-bc4a5a2e147b1d8746cef31b1abbd56425cdc3191af234c7477a90d3e2cfa992 WatchSource:0}: Error finding container bc4a5a2e147b1d8746cef31b1abbd56425cdc3191af234c7477a90d3e2cfa992: Status 404 returned error can't find the container with id bc4a5a2e147b1d8746cef31b1abbd56425cdc3191af234c7477a90d3e2cfa992 Sep 07 13:01:17.317904 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.317866 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" event={"ID":"06b6ce26-951d-44da-ae1d-4b20971557fd","Type":"ContainerStarted","Data":"6b30a90d19206242f53252d01f6271fc3b10281462b55864c9d30d95425d2508"} Sep 07 13:01:17.319588 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.319556 2565 generic.go:356] "Generic (PLEG): container finished" podID="22a97db3-4fcb-4e13-b1cc-54ffd9845248" containerID="1d78a3d661d540511e30e8d66cc98f7de2c05e8a4bd03e11c9e115b968a3e4b3" exitCode=0 Sep 07 13:01:17.319723 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.319631 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7lsw4" event={"ID":"22a97db3-4fcb-4e13-b1cc-54ffd9845248","Type":"ContainerDied","Data":"1d78a3d661d540511e30e8d66cc98f7de2c05e8a4bd03e11c9e115b968a3e4b3"} Sep 07 13:01:17.320754 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.320735 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" event={"ID":"aee2d72e-4493-453b-9be4-5b60e710cd2c","Type":"ContainerStarted","Data":"bc4a5a2e147b1d8746cef31b1abbd56425cdc3191af234c7477a90d3e2cfa992"} Sep 07 13:01:17.340858 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.340808 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-64j7z" podStartSLOduration=2.348125848 podStartE2EDuration="4.34079396s" totalImagesPullingTime="1.992668112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:01:17.338774551 +0000 UTC m=+99.019160347" watchObservedRunningTime="2026-09-07 13:01:17.34079396 +0000 UTC m=+99.021179730" Sep 07 13:01:17.761131 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.761096 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv"] Sep 07 13:01:17.781583 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.781556 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" Sep 07 13:01:17.784484 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.784455 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv"] Sep 07 13:01:17.786833 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.786810 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 07 13:01:17.787337 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.787313 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bjsqg\"" Sep 07 13:01:17.798534 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.798512 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7c797d6c-2sxp6"] Sep 07 13:01:17.823111 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.823082 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.826273 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.826253 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 07 13:01:17.826757 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.826738 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 07 13:01:17.827046 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.827030 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rdx7q\"" Sep 07 13:01:17.827331 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.827316 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 07 13:01:17.827331 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.827327 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dn6dpcuqkqds\"" Sep 07 13:01:17.829194 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.829176 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 07 13:01:17.833849 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.833828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78bm\" (UniqueName: \"kubernetes.io/projected/e7fa1752-9763-4427-945a-81a822d186bd-kube-api-access-r78bm\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.833950 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.833864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7fa1752-9763-4427-945a-81a822d186bd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.833950 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.833891 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10ae1775-1aa5-4079-b465-4945352e2cde-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-6zcxv\" (UID: \"10ae1775-1aa5-4079-b465-4945352e2cde\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" Sep 07 13:01:17.833950 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.833920 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e7fa1752-9763-4427-945a-81a822d186bd-secret-metrics-server-client-certs\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.834068 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.833949 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e7fa1752-9763-4427-945a-81a822d186bd-client-ca-bundle\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.834068 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.833975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxsj8\" (UniqueName: \"kubernetes.io/projected/10ae1775-1aa5-4079-b465-4945352e2cde-kube-api-access-wxsj8\") pod \"monitoring-plugin-85bbbbbf4-6zcxv\" (UID: \"10ae1775-1aa5-4079-b465-4945352e2cde\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" Sep 07 13:01:17.834068 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.834019 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e7fa1752-9763-4427-945a-81a822d186bd-metrics-server-audit-profiles\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.834068 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.834052 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e7fa1752-9763-4427-945a-81a822d186bd-audit-log\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.834202 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.834075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e7fa1752-9763-4427-945a-81a822d186bd-secret-metrics-server-tls\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.851743 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.851717 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c797d6c-2sxp6"] Sep 07 13:01:17.934643 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.934610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e7fa1752-9763-4427-945a-81a822d186bd-secret-metrics-server-tls\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.934822 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.934669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78bm\" (UniqueName: \"kubernetes.io/projected/e7fa1752-9763-4427-945a-81a822d186bd-kube-api-access-r78bm\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.934822 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.934694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7fa1752-9763-4427-945a-81a822d186bd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.934822 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.934714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10ae1775-1aa5-4079-b465-4945352e2cde-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-6zcxv\" (UID: \"10ae1775-1aa5-4079-b465-4945352e2cde\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" Sep 07 13:01:17.934822 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.934731 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e7fa1752-9763-4427-945a-81a822d186bd-secret-metrics-server-client-certs\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.934822 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.934751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e7fa1752-9763-4427-945a-81a822d186bd-client-ca-bundle\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.935075 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.934814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsj8\" (UniqueName: \"kubernetes.io/projected/10ae1775-1aa5-4079-b465-4945352e2cde-kube-api-access-wxsj8\") pod \"monitoring-plugin-85bbbbbf4-6zcxv\" (UID: \"10ae1775-1aa5-4079-b465-4945352e2cde\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" Sep 07 13:01:17.935075 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.934864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e7fa1752-9763-4427-945a-81a822d186bd-metrics-server-audit-profiles\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.935075 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.934901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e7fa1752-9763-4427-945a-81a822d186bd-audit-log\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.935346 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.935300 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e7fa1752-9763-4427-945a-81a822d186bd-audit-log\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.935579 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.935554 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7fa1752-9763-4427-945a-81a822d186bd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.935945 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.935927 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e7fa1752-9763-4427-945a-81a822d186bd-metrics-server-audit-profiles\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.937858 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.937838 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e7fa1752-9763-4427-945a-81a822d186bd-secret-metrics-server-client-certs\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.937914 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.937856 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e7fa1752-9763-4427-945a-81a822d186bd-secret-metrics-server-tls\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.937914 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.937900 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e7fa1752-9763-4427-945a-81a822d186bd-client-ca-bundle\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.938361 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.938343 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10ae1775-1aa5-4079-b465-4945352e2cde-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-6zcxv\" (UID: \"10ae1775-1aa5-4079-b465-4945352e2cde\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" Sep 07 13:01:17.944116 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.944098 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78bm\" (UniqueName: \"kubernetes.io/projected/e7fa1752-9763-4427-945a-81a822d186bd-kube-api-access-r78bm\") pod \"metrics-server-7c797d6c-2sxp6\" (UID: \"e7fa1752-9763-4427-945a-81a822d186bd\") " pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:17.944255 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:17.944238 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxsj8\" (UniqueName: \"kubernetes.io/projected/10ae1775-1aa5-4079-b465-4945352e2cde-kube-api-access-wxsj8\") pod \"monitoring-plugin-85bbbbbf4-6zcxv\" (UID: \"10ae1775-1aa5-4079-b465-4945352e2cde\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" Sep 07 13:01:18.091024 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:18.090934 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" Sep 07 13:01:18.132407 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:18.132381 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:18.327121 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:18.327082 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7lsw4" event={"ID":"22a97db3-4fcb-4e13-b1cc-54ffd9845248","Type":"ContainerStarted","Data":"3bba6407875f3da1bd56ce4a046e5dec81e48346e5b48a67659a1dd98390eef0"} Sep 07 13:01:18.327518 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:18.327139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7lsw4" event={"ID":"22a97db3-4fcb-4e13-b1cc-54ffd9845248","Type":"ContainerStarted","Data":"de3cdd90f6bc67c07be53a74066fd68adfb62700c12a8fa0b6b501a02faaa048"} Sep 07 13:01:18.394936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:18.394828 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7lsw4" podStartSLOduration=2.098103858 podStartE2EDuration="5.39480984s" totalImagesPullingTime="3.217243603s" totalInitContainerRuntime="79.462379ms" isStatefulPod=true podCreationTimestamp="2026-09-07 13:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:01:18.354404191 +0000 UTC m=+100.034789962" watchObservedRunningTime="2026-09-07 13:01:18.39480984 +0000 UTC m=+100.075195615" Sep 07 13:01:18.395095 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:18.395062 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv"] Sep 07 13:01:18.397868 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:01:18.397841 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ae1775_1aa5_4079_b465_4945352e2cde.slice/crio-cd513243148b39cffc78eb59c9d28158050f49dedde2e6253f3b93415c6ae858 WatchSource:0}: Error finding container cd513243148b39cffc78eb59c9d28158050f49dedde2e6253f3b93415c6ae858: Status 404 returned error can't find the container with id cd513243148b39cffc78eb59c9d28158050f49dedde2e6253f3b93415c6ae858 Sep 07 13:01:18.444771 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:18.444741 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7c797d6c-2sxp6"] Sep 07 13:01:18.447638 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:01:18.447612 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fa1752_9763_4427_945a_81a822d186bd.slice/crio-9171bd64496b9e0fab9e548d97aef1fdec609468b39bf19fe85d6896b0b2dac9 WatchSource:0}: Error finding container 9171bd64496b9e0fab9e548d97aef1fdec609468b39bf19fe85d6896b0b2dac9: Status 404 returned error can't find the container with id 9171bd64496b9e0fab9e548d97aef1fdec609468b39bf19fe85d6896b0b2dac9 Sep 07 13:01:19.334339 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.334223 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" event={"ID":"e7fa1752-9763-4427-945a-81a822d186bd","Type":"ContainerStarted","Data":"9171bd64496b9e0fab9e548d97aef1fdec609468b39bf19fe85d6896b0b2dac9"} Sep 07 13:01:19.337129 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.337090 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" event={"ID":"10ae1775-1aa5-4079-b465-4945352e2cde","Type":"ContainerStarted","Data":"cd513243148b39cffc78eb59c9d28158050f49dedde2e6253f3b93415c6ae858"} Sep 07 13:01:19.429577 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.429542 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 13:01:19.448669 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.448628 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.454330 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.453931 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 07 13:01:19.454713 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.454352 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 07 13:01:19.454713 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.454546 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 07 13:01:19.454864 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.454739 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7iqmuc3l4cbg8\"" Sep 07 13:01:19.454954 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.454936 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 07 13:01:19.455369 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.455216 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 07 13:01:19.455477 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.455458 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 07 13:01:19.456771 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.455902 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 07 13:01:19.456771 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.455918 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k4vwh\"" Sep 07 13:01:19.456771 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.456160 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 07 13:01:19.456771 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.456173 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 07 13:01:19.456771 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.456411 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 07 13:01:19.457463 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.457202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 13:01:19.462530 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.461633 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 07 13:01:19.464273 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.464131 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 07 13:01:19.465394 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.464755 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 07 13:01:19.467120 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.467101 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 07 13:01:19.547369 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hngp2\" (UniqueName: \"kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-kube-api-access-hngp2\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547564 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547387 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547564 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547435 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547564 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547564 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547517 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547564 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547546 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547608 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547649 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-config\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547734 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547754 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547774 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.547863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.548275 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.548275 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.548275 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.547975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.548275 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.548002 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.548275 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.548028 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.548275 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.548053 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.648990 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.648951 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649163 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649226 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649286 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649239 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649338 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-config\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649338 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649432 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649432 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649384 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649432 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649580 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649580 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649496 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649580 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649542 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649580 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649808 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649808 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649808 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649808 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649808 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649730 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hngp2\" (UniqueName: \"kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-kube-api-access-hngp2\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649808 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.649808 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649798 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.650131 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.649973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.650131 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.650010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.650782 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.650716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.651024 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.650996 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.651143 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.651045 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.651412 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.651389 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.651685 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.651588 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.652583 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.652136 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.653501 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.653459 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.655825 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.654272 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.655825 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.654973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.655825 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.654976 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.655825 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.655500 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.656621 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.656273 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-web-config\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.657490 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.657465 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.657838 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.657791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.660913 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.660886 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.661095 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.661077 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-config\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.661313 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.661280 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-config-out\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.663714 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.663673 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hngp2\" (UniqueName: \"kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-kube-api-access-hngp2\") pod \"prometheus-k8s-0\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:19.767350 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:19.767295 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:20.083794 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:20.083709 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 13:01:20.328161 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:01:20.328123 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda242c342_a828_49e8_a1ba_0333e92a07a8.slice/crio-9b7fad4185ee356eb3b5b569dfb45595ff0ae0a92871d6c467faa2993174a07c WatchSource:0}: Error finding container 9b7fad4185ee356eb3b5b569dfb45595ff0ae0a92871d6c467faa2993174a07c: Status 404 returned error can't find the container with id 9b7fad4185ee356eb3b5b569dfb45595ff0ae0a92871d6c467faa2993174a07c Sep 07 13:01:20.341161 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:20.341098 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerStarted","Data":"9b7fad4185ee356eb3b5b569dfb45595ff0ae0a92871d6c467faa2993174a07c"} Sep 07 13:01:22.352366 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:22.352266 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" event={"ID":"aee2d72e-4493-453b-9be4-5b60e710cd2c","Type":"ContainerStarted","Data":"37de16e050acc8af0d4c54eef3b53bf80dcb029be7928d3c6433d105fc488c29"} Sep 07 13:01:22.352855 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:22.352369 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" event={"ID":"aee2d72e-4493-453b-9be4-5b60e710cd2c","Type":"ContainerStarted","Data":"f9c29dfa17bfb7b6caa15aede7986723c81f04a980fe125a9aada280fe0d750b"} Sep 07 13:01:22.354135 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:22.354103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" event={"ID":"e7fa1752-9763-4427-945a-81a822d186bd","Type":"ContainerStarted","Data":"cd48ea5fe8e9b7ae5c011ab2d03c550793d28e6214e6f157b76dc65f4c3edbe5"} Sep 07 13:01:22.355935 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:22.355910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" event={"ID":"10ae1775-1aa5-4079-b465-4945352e2cde","Type":"ContainerStarted","Data":"44d754df4f8a28a059d9a44c59306c55fde6cf2907cedabcbaa57936b5399b8e"} Sep 07 13:01:22.356136 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:22.356109 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" Sep 07 13:01:22.361705 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:22.361673 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" Sep 07 13:01:22.375131 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:22.375069 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" podStartSLOduration=1.9270339170000002 podStartE2EDuration="5.375049584s" totalImagesPullingTime="3.448015667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:01:22.373264311 +0000 UTC m=+104.053650083" watchObservedRunningTime="2026-09-07 13:01:22.375049584 +0000 UTC m=+104.055435355" Sep 07 13:01:22.390708 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:22.390642 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-6zcxv" podStartSLOduration=1.892709982 podStartE2EDuration="5.390620477s" totalImagesPullingTime="3.497910495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:01:22.389822675 +0000 UTC m=+104.070208446" watchObservedRunningTime="2026-09-07 13:01:22.390620477 +0000 UTC m=+104.071006250" Sep 07 13:01:23.184670 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:23.184638 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tq7v6" Sep 07 13:01:23.363183 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:23.363147 2565 generic.go:356] "Generic (PLEG): container finished" podID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerID="70a1bb097fccd0e7d710153f8c4fd608837cafb3fa978fa45907c7a4652ae061" exitCode=0 Sep 07 13:01:23.363693 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:23.363246 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerDied","Data":"70a1bb097fccd0e7d710153f8c4fd608837cafb3fa978fa45907c7a4652ae061"} Sep 07 13:01:23.365906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:23.365877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" event={"ID":"aee2d72e-4493-453b-9be4-5b60e710cd2c","Type":"ContainerStarted","Data":"4b5eac832413783624d54d830e1c14dee2a157b97b8f2abf2b59074a75470c26"} Sep 07 13:01:24.377697 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:24.377525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" event={"ID":"aee2d72e-4493-453b-9be4-5b60e710cd2c","Type":"ContainerStarted","Data":"33735d4304e3d2cc4e1a0801379622c3d2bcf7b38aaf9fb34928f99e58bf47dd"} Sep 07 13:01:24.377697 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:24.377636 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" event={"ID":"aee2d72e-4493-453b-9be4-5b60e710cd2c","Type":"ContainerStarted","Data":"2e71ddfafa9244e8061a88aba4ab84d82d067f6b921dc533cd5f094057b5e49a"} Sep 07 13:01:25.385220 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:25.385181 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" event={"ID":"aee2d72e-4493-453b-9be4-5b60e710cd2c","Type":"ContainerStarted","Data":"526f5c011f3280b2a1648af0109a82f24342a85243a7b74635dc3990d0ad2506"} Sep 07 13:01:25.385759 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:25.385297 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:25.412385 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:25.412337 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" podStartSLOduration=2.9313116040000002 podStartE2EDuration="9.412310748s" totalImagesPullingTime="6.480999144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:01:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:01:25.411050844 +0000 UTC m=+107.091436614" watchObservedRunningTime="2026-09-07 13:01:25.412310748 +0000 UTC m=+107.092696517" Sep 07 13:01:27.401324 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:27.401286 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerStarted","Data":"e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176"} Sep 07 13:01:27.401791 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:27.401371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerStarted","Data":"1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635"} Sep 07 13:01:27.401791 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:27.401415 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerStarted","Data":"8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362"} Sep 07 13:01:28.408415 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:28.408379 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerStarted","Data":"f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094"} Sep 07 13:01:28.408902 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:28.408442 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerStarted","Data":"ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8"} Sep 07 13:01:28.408902 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:28.408468 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerStarted","Data":"2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89"} Sep 07 13:01:28.457176 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:28.457106 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.28038081 podStartE2EDuration="9.457081947s" totalImagesPullingTime="6.160460192s" totalInitContainerRuntime="16.240945ms" isStatefulPod=false podCreationTimestamp="2026-09-07 13:01:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:01:28.453657484 +0000 UTC m=+110.134043253" watchObservedRunningTime="2026-09-07 13:01:28.457081947 +0000 UTC m=+110.137467719" Sep 07 13:01:29.768188 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:29.768150 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:01:31.400570 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:31.400541 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f656656f-7r4lc" Sep 07 13:01:38.133133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:38.133084 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:38.133673 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:38.133178 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:58.138835 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:58.138804 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:01:58.142696 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:01:58.142670 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7c797d6c-2sxp6" Sep 07 13:02:19.768454 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:19.768414 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:19.787170 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:19.787141 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:20.645520 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:20.645495 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:38.066733 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.066653 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 13:02:38.067133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.066968 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="prometheus" containerID="cri-o://8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362" gracePeriod=600 Sep 07 13:02:38.067133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.067011 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094" gracePeriod=600 Sep 07 13:02:38.067133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.067027 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy-web" containerID="cri-o://2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89" gracePeriod=600 Sep 07 13:02:38.067133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.067041 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="thanos-sidecar" containerID="cri-o://e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176" gracePeriod=600 Sep 07 13:02:38.067133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.066987 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy" containerID="cri-o://ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8" gracePeriod=600 Sep 07 13:02:38.067389 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.067048 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="config-reloader" containerID="cri-o://1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635" gracePeriod=600 Sep 07 13:02:38.705303 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.705269 2565 generic.go:356] "Generic (PLEG): container finished" podID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerID="2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89" exitCode=0 Sep 07 13:02:38.705303 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.705296 2565 generic.go:356] "Generic (PLEG): container finished" podID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerID="e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176" exitCode=0 Sep 07 13:02:38.705303 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.705307 2565 generic.go:356] "Generic (PLEG): container finished" podID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerID="1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635" exitCode=0 Sep 07 13:02:38.705608 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.705346 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerDied","Data":"2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89"} Sep 07 13:02:38.705608 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.705406 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerDied","Data":"e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176"} Sep 07 13:02:38.705608 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:38.705432 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerDied","Data":"1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635"} Sep 07 13:02:39.712502 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:39.712471 2565 generic.go:356] "Generic (PLEG): container finished" podID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerID="f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094" exitCode=0 Sep 07 13:02:39.712502 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:39.712500 2565 generic.go:356] "Generic (PLEG): container finished" podID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerID="ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8" exitCode=0 Sep 07 13:02:39.712948 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:39.712524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerDied","Data":"f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094"} Sep 07 13:02:39.712948 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:39.712606 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerDied","Data":"ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8"} Sep 07 13:02:40.632132 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:02:40.632092 2565 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Sep 07 13:02:40.633087 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:02:40.633042 2565 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Sep 07 13:02:40.633934 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:02:40.633906 2565 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Sep 07 13:02:40.634029 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:02:40.633947 2565 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="prometheus" probeResult="unknown" Sep 07 13:02:45.407190 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.407167 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.516354 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516273 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.516354 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516306 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.516354 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516336 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-serving-certs-ca-bundle\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.516638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516369 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-kubelet-serving-ca-bundle\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.516638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516397 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hngp2\" (UniqueName: \"kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-kube-api-access-hngp2\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.516638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516451 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-kube-rbac-proxy\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.516638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516475 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-grpc-tls\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.516638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516502 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-tls\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.516638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516531 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-0\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.516638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516557 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-metrics-client-certs\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.516638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516585 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-thanos-prometheus-http-client-file\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.516638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516629 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-config\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.517088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516656 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-1\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.517088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516686 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-tls-assets\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.517088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516715 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-web-config\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.517088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516744 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-config-out\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.517088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516764 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:02:45.517088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516777 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-db\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.517088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516841 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-trusted-ca-bundle\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.517088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.516873 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-metrics-client-ca\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.517088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.517000 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-2\") pod \"a242c342-a828-49e8-a1ba-0333e92a07a8\" (UID: \"a242c342-a828-49e8-a1ba-0333e92a07a8\") " Sep 07 13:02:45.517088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.517009 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:02:45.517623 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.517265 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.517623 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.517285 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.517623 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.517418 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:02:45.517623 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.517551 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:02:45.518148 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.517889 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:02:45.518148 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.518110 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:02:45.518511 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.518489 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:02:45.519272 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.519026 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:02:45.519272 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.519120 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:02:45.519425 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.519367 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-config" (OuterVolumeSpecName: "config") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:02:45.521242 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.520918 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:02:45.521242 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.520939 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:02:45.521242 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.521003 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:02:45.521242 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.521025 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-kube-api-access-hngp2" (OuterVolumeSpecName: "kube-api-access-hngp2") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "kube-api-access-hngp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:02:45.521456 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.521396 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:02:45.521730 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.521712 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:02:45.521987 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.521965 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:02:45.522125 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.522107 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:02:45.523020 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.523005 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-config-out" (OuterVolumeSpecName: "config-out") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:02:45.532313 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.532290 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-web-config" (OuterVolumeSpecName: "web-config") pod "a242c342-a828-49e8-a1ba-0333e92a07a8" (UID: "a242c342-a828-49e8-a1ba-0333e92a07a8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:02:45.617871 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617840 2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-kube-rbac-proxy\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.617871 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617866 2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-grpc-tls\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.617871 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617875 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617884 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617892 2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-metrics-client-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617901 2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617911 2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-config\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617918 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617926 2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-tls-assets\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617934 2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-web-config\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617941 2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-config-out\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617948 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-db\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617956 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617964 2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-configmap-metrics-client-ca\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617971 2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a242c342-a828-49e8-a1ba-0333e92a07a8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617981 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617990 2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a242c342-a828-49e8-a1ba-0333e92a07a8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.618064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.617998 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hngp2\" (UniqueName: \"kubernetes.io/projected/a242c342-a828-49e8-a1ba-0333e92a07a8-kube-api-access-hngp2\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:02:45.740358 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.740322 2565 generic.go:356] "Generic (PLEG): container finished" podID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerID="8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362" exitCode=0 Sep 07 13:02:45.740502 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.740398 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.740502 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.740402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerDied","Data":"8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362"} Sep 07 13:02:45.740502 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.740472 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a242c342-a828-49e8-a1ba-0333e92a07a8","Type":"ContainerDied","Data":"9b7fad4185ee356eb3b5b569dfb45595ff0ae0a92871d6c467faa2993174a07c"} Sep 07 13:02:45.740648 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.740505 2565 scope.go:117] "RemoveContainer" containerID="f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094" Sep 07 13:02:45.748555 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.748535 2565 scope.go:117] "RemoveContainer" containerID="ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8" Sep 07 13:02:45.755805 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.755788 2565 scope.go:117] "RemoveContainer" containerID="2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89" Sep 07 13:02:45.762144 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.762127 2565 scope.go:117] "RemoveContainer" containerID="e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176" Sep 07 13:02:45.766418 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.766344 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 13:02:45.769647 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.769623 2565 scope.go:117] "RemoveContainer" containerID="1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635" Sep 07 13:02:45.771265 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.771246 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 13:02:45.776359 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.776344 2565 scope.go:117] "RemoveContainer" containerID="8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362" Sep 07 13:02:45.783434 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.783416 2565 scope.go:117] "RemoveContainer" containerID="70a1bb097fccd0e7d710153f8c4fd608837cafb3fa978fa45907c7a4652ae061" Sep 07 13:02:45.789855 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.789837 2565 scope.go:117] "RemoveContainer" containerID="f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094" Sep 07 13:02:45.790106 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:02:45.790089 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094\": container with ID starting with f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094 not found: ID does not exist" containerID="f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094" Sep 07 13:02:45.790157 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.790114 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094"} err="failed to get container status \"f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094\": rpc error: code = NotFound desc = could not find container \"f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094\": container with ID starting with f53eaf2388b102e8a4974c24e6effce18c48954b9650f1a993d424065e82f094 not found: ID does not exist" Sep 07 13:02:45.790157 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.790146 2565 scope.go:117] "RemoveContainer" containerID="ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8" Sep 07 13:02:45.790385 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:02:45.790368 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8\": container with ID starting with ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8 not found: ID does not exist" containerID="ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8" Sep 07 13:02:45.790429 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.790388 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8"} err="failed to get container status \"ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8\": rpc error: code = NotFound desc = could not find container \"ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8\": container with ID starting with ae8a1a7355a1b2f5d856a9c89bbeeac8859ed9ba6a24ed17aae87d97ef5100f8 not found: ID does not exist" Sep 07 13:02:45.790429 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.790407 2565 scope.go:117] "RemoveContainer" containerID="2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89" Sep 07 13:02:45.790624 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:02:45.790585 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89\": container with ID starting with 2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89 not found: ID does not exist" containerID="2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89" Sep 07 13:02:45.790670 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.790633 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89"} err="failed to get container status \"2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89\": rpc error: code = NotFound desc = could not find container \"2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89\": container with ID starting with 2fab7603a4d5d9c9fcabd84f4f05965420ca38edb4a950391eb52e5f14b71b89 not found: ID does not exist" Sep 07 13:02:45.790670 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.790655 2565 scope.go:117] "RemoveContainer" containerID="e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176" Sep 07 13:02:45.790910 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:02:45.790892 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176\": container with ID starting with e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176 not found: ID does not exist" containerID="e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176" Sep 07 13:02:45.790979 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.790919 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176"} err="failed to get container status \"e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176\": rpc error: code = NotFound desc = could not find container \"e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176\": container with ID starting with e1371b45c88059675267e1135e8c12d7b04c1b22b04e00c49ead7690567ff176 not found: ID does not exist" Sep 07 13:02:45.790979 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.790940 2565 scope.go:117] "RemoveContainer" containerID="1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635" Sep 07 13:02:45.791177 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:02:45.791161 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635\": container with ID starting with 1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635 not found: ID does not exist" containerID="1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635" Sep 07 13:02:45.791238 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.791184 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635"} err="failed to get container status \"1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635\": rpc error: code = NotFound desc = could not find container \"1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635\": container with ID starting with 1ab8b38444ef3519cc344221f981a1f8217ca9fe71526e8f0e5f87578a1ad635 not found: ID does not exist" Sep 07 13:02:45.791238 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.791202 2565 scope.go:117] "RemoveContainer" containerID="8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362" Sep 07 13:02:45.791427 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:02:45.791413 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362\": container with ID starting with 8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362 not found: ID does not exist" containerID="8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362" Sep 07 13:02:45.791484 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.791432 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362"} err="failed to get container status \"8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362\": rpc error: code = NotFound desc = could not find container \"8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362\": container with ID starting with 8175c89903ab9bbb564745f20e060e6687551693e481b4a6f6a7542951ca0362 not found: ID does not exist" Sep 07 13:02:45.791484 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.791449 2565 scope.go:117] "RemoveContainer" containerID="70a1bb097fccd0e7d710153f8c4fd608837cafb3fa978fa45907c7a4652ae061" Sep 07 13:02:45.791705 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:02:45.791686 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a1bb097fccd0e7d710153f8c4fd608837cafb3fa978fa45907c7a4652ae061\": container with ID starting with 70a1bb097fccd0e7d710153f8c4fd608837cafb3fa978fa45907c7a4652ae061 not found: ID does not exist" containerID="70a1bb097fccd0e7d710153f8c4fd608837cafb3fa978fa45907c7a4652ae061" Sep 07 13:02:45.791775 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.791709 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a1bb097fccd0e7d710153f8c4fd608837cafb3fa978fa45907c7a4652ae061"} err="failed to get container status \"70a1bb097fccd0e7d710153f8c4fd608837cafb3fa978fa45907c7a4652ae061\": rpc error: code = NotFound desc = could not find container \"70a1bb097fccd0e7d710153f8c4fd608837cafb3fa978fa45907c7a4652ae061\": container with ID starting with 70a1bb097fccd0e7d710153f8c4fd608837cafb3fa978fa45907c7a4652ae061 not found: ID does not exist" Sep 07 13:02:45.808531 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808511 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 13:02:45.808812 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808800 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="init-config-reloader" Sep 07 13:02:45.808859 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808814 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="init-config-reloader" Sep 07 13:02:45.808859 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808824 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy-web" Sep 07 13:02:45.808859 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808831 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy-web" Sep 07 13:02:45.808859 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808842 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy" Sep 07 13:02:45.808859 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808848 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy" Sep 07 13:02:45.808859 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808855 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="config-reloader" Sep 07 13:02:45.808859 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808860 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="config-reloader" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808866 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy-thanos" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808871 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy-thanos" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808880 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="prometheus" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808884 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="prometheus" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808891 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="thanos-sidecar" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808896 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="thanos-sidecar" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808966 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808973 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy-thanos" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808981 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="thanos-sidecar" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808986 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="kube-rbac-proxy-web" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808992 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="prometheus" Sep 07 13:02:45.809048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.808998 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" containerName="config-reloader" Sep 07 13:02:45.812462 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.812446 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.815432 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.815409 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 07 13:02:45.816416 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.816388 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 07 13:02:45.816510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.816392 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 07 13:02:45.816648 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.816627 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k4vwh\"" Sep 07 13:02:45.816756 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.816725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 07 13:02:45.816824 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.816778 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 07 13:02:45.816824 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.816806 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7iqmuc3l4cbg8\"" Sep 07 13:02:45.817022 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.817004 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 07 13:02:45.817089 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.817030 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 07 13:02:45.817679 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.817663 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 07 13:02:45.817883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.817868 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 07 13:02:45.818510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.818302 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 07 13:02:45.818510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.818369 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 07 13:02:45.819932 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.819912 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 07 13:02:45.821723 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.821701 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 07 13:02:45.823270 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.823254 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 07 13:02:45.859527 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.859349 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 13:02:45.919882 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.919848 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-config\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.919882 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.919880 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40429edf-7358-4009-8a8e-76aa05defb1b-config-out\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920067 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.919900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920067 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.919975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920067 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920016 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920067 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r22vz\" (UniqueName: \"kubernetes.io/projected/40429edf-7358-4009-8a8e-76aa05defb1b-kube-api-access-r22vz\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920067 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920053 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920232 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920090 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920232 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920113 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920232 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920136 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-web-config\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920232 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920173 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920232 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920192 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920382 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920256 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920382 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920382 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920301 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40429edf-7358-4009-8a8e-76aa05defb1b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920382 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920329 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920382 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920348 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920382 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920366 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920553 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920392 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:45.920553 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:45.920409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.021804 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.021707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.021804 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.021760 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-web-config\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.021804 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.021786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.021804 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.021803 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.022133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.021826 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.022133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.021851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.022133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.021875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40429edf-7358-4009-8a8e-76aa05defb1b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.022133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.021908 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.022133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.021942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.022386 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.022137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.022498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.022561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.022610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023044 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.022561 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023139 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-config\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40429edf-7358-4009-8a8e-76aa05defb1b-config-out\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023211 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023248 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023249 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023278 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023308 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023342 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r22vz\" (UniqueName: \"kubernetes.io/projected/40429edf-7358-4009-8a8e-76aa05defb1b-kube-api-access-r22vz\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.023996 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.023461 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.024937 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.024766 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.025477 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.025418 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-web-config\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.025580 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.025541 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.025580 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.025541 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.025978 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.025948 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-config\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.026335 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.026315 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40429edf-7358-4009-8a8e-76aa05defb1b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.026939 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.026920 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.027010 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.026950 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.027470 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.027442 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.027618 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.027574 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40429edf-7358-4009-8a8e-76aa05defb1b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.027618 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.027577 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.027719 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.027663 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40429edf-7358-4009-8a8e-76aa05defb1b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.028037 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.028020 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40429edf-7358-4009-8a8e-76aa05defb1b-config-out\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.034940 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.034923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r22vz\" (UniqueName: \"kubernetes.io/projected/40429edf-7358-4009-8a8e-76aa05defb1b-kube-api-access-r22vz\") pod \"prometheus-k8s-0\" (UID: \"40429edf-7358-4009-8a8e-76aa05defb1b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.122775 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.122749 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:02:46.408649 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.408622 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 13:02:46.410555 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:02:46.410530 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40429edf_7358_4009_8a8e_76aa05defb1b.slice/crio-7e92203d3c0ff02581bbedefb308f931a0b76a17a5ce8611541a12200fb9666a WatchSource:0}: Error finding container 7e92203d3c0ff02581bbedefb308f931a0b76a17a5ce8611541a12200fb9666a: Status 404 returned error can't find the container with id 7e92203d3c0ff02581bbedefb308f931a0b76a17a5ce8611541a12200fb9666a Sep 07 13:02:46.746838 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.746753 2565 generic.go:356] "Generic (PLEG): container finished" podID="40429edf-7358-4009-8a8e-76aa05defb1b" containerID="63a3affe61206b4e59b4f555ecd9873d01060651f833d3fade06b0887ba8283b" exitCode=0 Sep 07 13:02:46.747052 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.746845 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40429edf-7358-4009-8a8e-76aa05defb1b","Type":"ContainerDied","Data":"63a3affe61206b4e59b4f555ecd9873d01060651f833d3fade06b0887ba8283b"} Sep 07 13:02:46.747052 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.746906 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40429edf-7358-4009-8a8e-76aa05defb1b","Type":"ContainerStarted","Data":"7e92203d3c0ff02581bbedefb308f931a0b76a17a5ce8611541a12200fb9666a"} Sep 07 13:02:46.826663 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:46.826638 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a242c342-a828-49e8-a1ba-0333e92a07a8" path="/var/lib/kubelet/pods/a242c342-a828-49e8-a1ba-0333e92a07a8/volumes" Sep 07 13:02:47.755173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:47.755139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40429edf-7358-4009-8a8e-76aa05defb1b","Type":"ContainerStarted","Data":"68f518614b4a140ce2b7119af245bc42e5fe4a1ddd6373a6227aad4a72e2a00c"} Sep 07 13:02:47.755631 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:47.755200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40429edf-7358-4009-8a8e-76aa05defb1b","Type":"ContainerStarted","Data":"7ae1eeb5cb739a92e83a698721c07c65e38025c1af7ad5f2137cb7458ce5ed84"} Sep 07 13:02:47.755631 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:47.755227 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40429edf-7358-4009-8a8e-76aa05defb1b","Type":"ContainerStarted","Data":"4558335f363f7dac51cd5633d8ceb5088be4a49eb1f6ed73e3418dec34784bea"} Sep 07 13:02:47.755631 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:47.755258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40429edf-7358-4009-8a8e-76aa05defb1b","Type":"ContainerStarted","Data":"d5bd25e97c66fc7b614d9e27104c62c828873340ed79d5bff39fcebb62f2c740"} Sep 07 13:02:47.755631 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:47.755281 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40429edf-7358-4009-8a8e-76aa05defb1b","Type":"ContainerStarted","Data":"8dcccd7f52a6a3f2eed6825eb0b405246c48b96b26db49b1560dad25162bbcb9"} Sep 07 13:02:47.755631 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:47.755304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40429edf-7358-4009-8a8e-76aa05defb1b","Type":"ContainerStarted","Data":"14184b6fba69dc5fc291a35e1fcf899a42012671995d1a9181d49eec17e55c84"} Sep 07 13:02:47.797501 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:47.797427 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.778271293 podStartE2EDuration="2.797412055s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.140762ms" isStatefulPod=false podCreationTimestamp="2026-09-07 13:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:02:47.793921554 +0000 UTC m=+189.474307323" watchObservedRunningTime="2026-09-07 13:02:47.797412055 +0000 UTC m=+189.477797825" Sep 07 13:02:51.123541 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:02:51.123499 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:03:46.123367 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:03:46.123333 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:03:46.138375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:03:46.138350 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:03:46.992176 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:03:46.992141 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 13:04:38.750103 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:04:38.750074 2565 kubelet.go:1648] "Image garbage collection succeeded" Sep 07 13:06:13.163710 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.163625 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl"] Sep 07 13:06:13.166444 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.166428 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.169415 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.169390 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 07 13:06:13.169535 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.169497 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 07 13:06:13.169926 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.169909 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 07 13:06:13.170007 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.169980 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vvqfs\"" Sep 07 13:06:13.170222 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.170203 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 07 13:06:13.190699 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.190666 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl"] Sep 07 13:06:13.248300 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.248264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-webhook-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.248300 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.248303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-manifests\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.248556 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.248320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfnvt\" (UniqueName: \"kubernetes.io/projected/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-kube-api-access-kfnvt\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.248556 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.248431 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-tmp\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.248556 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.248447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.349568 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.349534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-webhook-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.349748 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.349574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-manifests\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.349748 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.349614 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfnvt\" (UniqueName: \"kubernetes.io/projected/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-kube-api-access-kfnvt\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.349748 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.349675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-tmp\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.349748 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.349692 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.350053 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.350028 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-manifests\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.352020 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.351992 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-tmp\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.352164 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.352150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-webhook-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.352223 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.352178 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.362901 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.362874 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfnvt\" (UniqueName: \"kubernetes.io/projected/2eae9c0e-b88a-4a38-9a5c-e58c2c744c38-kube-api-access-kfnvt\") pod \"opendatahub-operator-controller-manager-5545ff49d6-j24tl\" (UID: \"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38\") " pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.476775 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.476703 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:13.789274 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.789249 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl"] Sep 07 13:06:13.791951 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:06:13.791926 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eae9c0e_b88a_4a38_9a5c_e58c2c744c38.slice/crio-ca890be2ad4553841fcf83756185c89409310f10375c4383ed8c2982ed3fbc6e WatchSource:0}: Error finding container ca890be2ad4553841fcf83756185c89409310f10375c4383ed8c2982ed3fbc6e: Status 404 returned error can't find the container with id ca890be2ad4553841fcf83756185c89409310f10375c4383ed8c2982ed3fbc6e Sep 07 13:06:13.793645 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:13.793630 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 13:06:14.533005 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:14.532964 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" event={"ID":"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38","Type":"ContainerStarted","Data":"ca890be2ad4553841fcf83756185c89409310f10375c4383ed8c2982ed3fbc6e"} Sep 07 13:06:17.548549 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:17.548517 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" event={"ID":"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38","Type":"ContainerStarted","Data":"4352ba4d54b72a18e4d4ce2a266aa8fec34d38b46d3e9a97aa9b8f3c1cdc3e15"} Sep 07 13:06:18.554078 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:18.554039 2565 generic.go:356] "Generic (PLEG): container finished" podID="2eae9c0e-b88a-4a38-9a5c-e58c2c744c38" containerID="4352ba4d54b72a18e4d4ce2a266aa8fec34d38b46d3e9a97aa9b8f3c1cdc3e15" exitCode=0 Sep 07 13:06:18.554518 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:18.554093 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" event={"ID":"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38","Type":"ContainerDied","Data":"4352ba4d54b72a18e4d4ce2a266aa8fec34d38b46d3e9a97aa9b8f3c1cdc3e15"} Sep 07 13:06:19.562048 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:19.561998 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" event={"ID":"2eae9c0e-b88a-4a38-9a5c-e58c2c744c38","Type":"ContainerStarted","Data":"e5d75f31bc91bce90e43179485ce14b5620f83dc0585b44155ece75986058a20"} Sep 07 13:06:19.562536 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:19.562093 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:19.599761 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:19.599708 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" podStartSLOduration=2.192720582 podStartE2EDuration="6.599688393s" totalImagesPullingTime="3.349306332s" totalInitContainerRuntime="1.057661479s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:19.599454467 +0000 UTC m=+401.279840239" watchObservedRunningTime="2026-09-07 13:06:19.599688393 +0000 UTC m=+401.280074164" Sep 07 13:06:30.569373 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:30.569342 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5545ff49d6-j24tl" Sep 07 13:06:32.251500 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.251462 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc"] Sep 07 13:06:32.254656 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.254640 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.257440 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.257417 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 07 13:06:32.258122 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.258106 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 07 13:06:32.258218 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.258142 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z6v8j\"" Sep 07 13:06:32.295095 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.295068 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 07 13:06:32.295706 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.295688 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 07 13:06:32.295801 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.295787 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 07 13:06:32.311694 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.311668 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc"] Sep 07 13:06:32.417524 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.417492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6d21826-fdba-448d-bb2f-4c20a3a1861c-cert\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.417720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.417568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6d21826-fdba-448d-bb2f-4c20a3a1861c-metrics-cert\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.417720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.417642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbvk6\" (UniqueName: \"kubernetes.io/projected/d6d21826-fdba-448d-bb2f-4c20a3a1861c-kube-api-access-dbvk6\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.417720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.417677 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6d21826-fdba-448d-bb2f-4c20a3a1861c-manager-config\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.518459 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.518384 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6d21826-fdba-448d-bb2f-4c20a3a1861c-metrics-cert\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.518459 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.518434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbvk6\" (UniqueName: \"kubernetes.io/projected/d6d21826-fdba-448d-bb2f-4c20a3a1861c-kube-api-access-dbvk6\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.518665 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.518464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6d21826-fdba-448d-bb2f-4c20a3a1861c-manager-config\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.518665 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.518496 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6d21826-fdba-448d-bb2f-4c20a3a1861c-cert\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.519357 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.519329 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6d21826-fdba-448d-bb2f-4c20a3a1861c-manager-config\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.521177 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.521150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6d21826-fdba-448d-bb2f-4c20a3a1861c-metrics-cert\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.521306 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.521285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6d21826-fdba-448d-bb2f-4c20a3a1861c-cert\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.532492 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.532466 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbvk6\" (UniqueName: \"kubernetes.io/projected/d6d21826-fdba-448d-bb2f-4c20a3a1861c-kube-api-access-dbvk6\") pod \"lws-controller-manager-58cbfb6566-4dcqc\" (UID: \"d6d21826-fdba-448d-bb2f-4c20a3a1861c\") " pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.563176 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.563153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:32.854171 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:32.854145 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc"] Sep 07 13:06:32.856190 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:06:32.856153 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d21826_fdba_448d_bb2f_4c20a3a1861c.slice/crio-052544b9a7ac34d804d2a4c2947c84007d0bac2da7f3d8a9a21683ae4d0ad40f WatchSource:0}: Error finding container 052544b9a7ac34d804d2a4c2947c84007d0bac2da7f3d8a9a21683ae4d0ad40f: Status 404 returned error can't find the container with id 052544b9a7ac34d804d2a4c2947c84007d0bac2da7f3d8a9a21683ae4d0ad40f Sep 07 13:06:33.587061 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.587028 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c"] Sep 07 13:06:33.591354 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.591332 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" Sep 07 13:06:33.600775 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.600755 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 07 13:06:33.600857 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.600824 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 07 13:06:33.600909 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.600869 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-52dkl\"" Sep 07 13:06:33.607979 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.607960 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 07 13:06:33.610183 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.610160 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 07 13:06:33.614289 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.614265 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c"] Sep 07 13:06:33.625509 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.625478 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" event={"ID":"d6d21826-fdba-448d-bb2f-4c20a3a1861c","Type":"ContainerStarted","Data":"052544b9a7ac34d804d2a4c2947c84007d0bac2da7f3d8a9a21683ae4d0ad40f"} Sep 07 13:06:33.730119 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.730084 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc20259-3152-445f-b813-f6dc5da42ed1-tls-certs\") pod \"kube-auth-proxy-546b89bbb7-spq8c\" (UID: \"2fc20259-3152-445f-b813-f6dc5da42ed1\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" Sep 07 13:06:33.730285 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.730125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fc20259-3152-445f-b813-f6dc5da42ed1-tmp\") pod \"kube-auth-proxy-546b89bbb7-spq8c\" (UID: \"2fc20259-3152-445f-b813-f6dc5da42ed1\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" Sep 07 13:06:33.730285 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.730156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r85nw\" (UniqueName: \"kubernetes.io/projected/2fc20259-3152-445f-b813-f6dc5da42ed1-kube-api-access-r85nw\") pod \"kube-auth-proxy-546b89bbb7-spq8c\" (UID: \"2fc20259-3152-445f-b813-f6dc5da42ed1\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" Sep 07 13:06:33.831239 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.831208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fc20259-3152-445f-b813-f6dc5da42ed1-tmp\") pod \"kube-auth-proxy-546b89bbb7-spq8c\" (UID: \"2fc20259-3152-445f-b813-f6dc5da42ed1\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" Sep 07 13:06:33.831412 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.831254 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r85nw\" (UniqueName: \"kubernetes.io/projected/2fc20259-3152-445f-b813-f6dc5da42ed1-kube-api-access-r85nw\") pod \"kube-auth-proxy-546b89bbb7-spq8c\" (UID: \"2fc20259-3152-445f-b813-f6dc5da42ed1\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" Sep 07 13:06:33.831412 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.831319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc20259-3152-445f-b813-f6dc5da42ed1-tls-certs\") pod \"kube-auth-proxy-546b89bbb7-spq8c\" (UID: \"2fc20259-3152-445f-b813-f6dc5da42ed1\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" Sep 07 13:06:33.833562 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.833530 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fc20259-3152-445f-b813-f6dc5da42ed1-tmp\") pod \"kube-auth-proxy-546b89bbb7-spq8c\" (UID: \"2fc20259-3152-445f-b813-f6dc5da42ed1\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" Sep 07 13:06:33.833779 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.833762 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc20259-3152-445f-b813-f6dc5da42ed1-tls-certs\") pod \"kube-auth-proxy-546b89bbb7-spq8c\" (UID: \"2fc20259-3152-445f-b813-f6dc5da42ed1\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" Sep 07 13:06:33.877054 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.876975 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r85nw\" (UniqueName: \"kubernetes.io/projected/2fc20259-3152-445f-b813-f6dc5da42ed1-kube-api-access-r85nw\") pod \"kube-auth-proxy-546b89bbb7-spq8c\" (UID: \"2fc20259-3152-445f-b813-f6dc5da42ed1\") " pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" Sep 07 13:06:33.900688 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:33.900658 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" Sep 07 13:06:34.206532 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:34.206492 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c"] Sep 07 13:06:34.210088 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:06:34.210057 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc20259_3152_445f_b813_f6dc5da42ed1.slice/crio-dc33fb18a074ea304283c13d693cfcc5ed47d82acf50bc649aaf2c616eecf6e9 WatchSource:0}: Error finding container dc33fb18a074ea304283c13d693cfcc5ed47d82acf50bc649aaf2c616eecf6e9: Status 404 returned error can't find the container with id dc33fb18a074ea304283c13d693cfcc5ed47d82acf50bc649aaf2c616eecf6e9 Sep 07 13:06:34.631748 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:34.631667 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" event={"ID":"2fc20259-3152-445f-b813-f6dc5da42ed1","Type":"ContainerStarted","Data":"dc33fb18a074ea304283c13d693cfcc5ed47d82acf50bc649aaf2c616eecf6e9"} Sep 07 13:06:37.647632 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:37.647513 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" event={"ID":"d6d21826-fdba-448d-bb2f-4c20a3a1861c","Type":"ContainerStarted","Data":"efebd36fdca6addcf1ea503b83f7ea72e34098d6768c049198cc5a32f1142e0d"} Sep 07 13:06:37.647632 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:37.647572 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:06:37.649141 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:37.649117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" event={"ID":"2fc20259-3152-445f-b813-f6dc5da42ed1","Type":"ContainerStarted","Data":"48125f15f6d6902eb6f89e832153f24c887e3f335347a64ffb53ec11696d925e"} Sep 07 13:06:37.667943 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:37.667895 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" podStartSLOduration=1.167254623 podStartE2EDuration="5.667882041s" totalImagesPullingTime="4.500627418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:37.666065683 +0000 UTC m=+419.346451453" watchObservedRunningTime="2026-09-07 13:06:37.667882041 +0000 UTC m=+419.348267811" Sep 07 13:06:37.685951 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:37.685893 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-546b89bbb7-spq8c" podStartSLOduration=1.491369079 podStartE2EDuration="4.685881143s" totalImagesPullingTime="3.194512064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:06:37.68478887 +0000 UTC m=+419.365174641" watchObservedRunningTime="2026-09-07 13:06:37.685881143 +0000 UTC m=+419.366266913" Sep 07 13:06:48.661383 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:06:48.661346 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58cbfb6566-4dcqc" Sep 07 13:08:38.344616 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:38.344515 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc"] Sep 07 13:08:38.347724 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:38.347709 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:38.350059 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:38.350041 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 07 13:08:38.350818 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:38.350802 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 07 13:08:38.350875 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:38.350835 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nszhg\"" Sep 07 13:08:38.378372 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:38.378346 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc"] Sep 07 13:08:38.462342 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:38.462314 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j28hm\" (UniqueName: \"kubernetes.io/projected/5f80882a-2987-41c0-822b-29c21fb5b622-kube-api-access-j28hm\") pod \"limitador-operator-controller-manager-6c549b6d85-fggpc\" (UID: \"5f80882a-2987-41c0-822b-29c21fb5b622\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:38.562777 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:38.562752 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j28hm\" (UniqueName: \"kubernetes.io/projected/5f80882a-2987-41c0-822b-29c21fb5b622-kube-api-access-j28hm\") pod \"limitador-operator-controller-manager-6c549b6d85-fggpc\" (UID: \"5f80882a-2987-41c0-822b-29c21fb5b622\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:38.572286 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:38.572261 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j28hm\" (UniqueName: \"kubernetes.io/projected/5f80882a-2987-41c0-822b-29c21fb5b622-kube-api-access-j28hm\") pod \"limitador-operator-controller-manager-6c549b6d85-fggpc\" (UID: \"5f80882a-2987-41c0-822b-29c21fb5b622\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:38.657407 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:38.657386 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:38.940749 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:38.940724 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc"] Sep 07 13:08:38.943239 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:08:38.943213 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f80882a_2987_41c0_822b_29c21fb5b622.slice/crio-4d0e4c2b4d29a35bdc13b2a38f6c2eb0403d28a5562847ea117db93aca8e88c1 WatchSource:0}: Error finding container 4d0e4c2b4d29a35bdc13b2a38f6c2eb0403d28a5562847ea117db93aca8e88c1: Status 404 returned error can't find the container with id 4d0e4c2b4d29a35bdc13b2a38f6c2eb0403d28a5562847ea117db93aca8e88c1 Sep 07 13:08:39.160255 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:39.160218 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" event={"ID":"5f80882a-2987-41c0-822b-29c21fb5b622","Type":"ContainerStarted","Data":"4d0e4c2b4d29a35bdc13b2a38f6c2eb0403d28a5562847ea117db93aca8e88c1"} Sep 07 13:08:42.175814 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:42.175778 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" event={"ID":"5f80882a-2987-41c0-822b-29c21fb5b622","Type":"ContainerStarted","Data":"cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107"} Sep 07 13:08:42.176270 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:42.175840 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:42.202340 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:42.202293 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" podStartSLOduration=1.79291367 podStartE2EDuration="4.202278438s" totalImagesPullingTime="2.409364768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:08:42.200697542 +0000 UTC m=+543.881083313" watchObservedRunningTime="2026-09-07 13:08:42.202278438 +0000 UTC m=+543.882664207" Sep 07 13:08:52.131525 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.131487 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc"] Sep 07 13:08:52.132003 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.131743 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" podUID="5f80882a-2987-41c0-822b-29c21fb5b622" containerName="manager" containerID="cri-o://cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107" gracePeriod=2 Sep 07 13:08:52.133677 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.133653 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:52.148221 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.148193 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc"] Sep 07 13:08:52.156803 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:08:52.156764 2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"5f80882a-2987-41c0-822b-29c21fb5b622\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-fggpc\": pods \"limitador-operator-controller-manager-6c549b6d85-fggpc\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:52.160391 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:08:52.160362 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fggpc\" is forbidden: User \"system:node:ip-10-0-142-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-216.ec2.internal' and this object" podUID="5f80882a-2987-41c0-822b-29c21fb5b622" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:52.176330 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.176304 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq"] Sep 07 13:08:52.176841 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.176824 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f80882a-2987-41c0-822b-29c21fb5b622" containerName="manager" Sep 07 13:08:52.176888 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.176845 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f80882a-2987-41c0-822b-29c21fb5b622" containerName="manager" Sep 07 13:08:52.176978 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.176956 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f80882a-2987-41c0-822b-29c21fb5b622" containerName="manager" Sep 07 13:08:52.180136 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.180120 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq" Sep 07 13:08:52.183133 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:08:52.183103 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fggpc\" is forbidden: User \"system:node:ip-10-0-142-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-216.ec2.internal' and this object" podUID="5f80882a-2987-41c0-822b-29c21fb5b622" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:52.200392 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.200369 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq"] Sep 07 13:08:52.266304 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.266277 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlj7m\" (UniqueName: \"kubernetes.io/projected/a6a5e5de-a0ca-45cf-b61d-c82ade4c22f4-kube-api-access-mlj7m\") pod \"limitador-operator-controller-manager-6c549b6d85-n42fq\" (UID: \"a6a5e5de-a0ca-45cf-b61d-c82ade4c22f4\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq" Sep 07 13:08:52.367234 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.367211 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlj7m\" (UniqueName: \"kubernetes.io/projected/a6a5e5de-a0ca-45cf-b61d-c82ade4c22f4-kube-api-access-mlj7m\") pod \"limitador-operator-controller-manager-6c549b6d85-n42fq\" (UID: \"a6a5e5de-a0ca-45cf-b61d-c82ade4c22f4\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq" Sep 07 13:08:52.370855 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.370837 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:52.373151 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:08:52.373129 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fggpc\" is forbidden: User \"system:node:ip-10-0-142-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-216.ec2.internal' and this object" podUID="5f80882a-2987-41c0-822b-29c21fb5b622" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:52.382794 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.382742 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlj7m\" (UniqueName: \"kubernetes.io/projected/a6a5e5de-a0ca-45cf-b61d-c82ade4c22f4-kube-api-access-mlj7m\") pod \"limitador-operator-controller-manager-6c549b6d85-n42fq\" (UID: \"a6a5e5de-a0ca-45cf-b61d-c82ade4c22f4\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq" Sep 07 13:08:52.468195 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.468169 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j28hm\" (UniqueName: \"kubernetes.io/projected/5f80882a-2987-41c0-822b-29c21fb5b622-kube-api-access-j28hm\") pod \"5f80882a-2987-41c0-822b-29c21fb5b622\" (UID: \"5f80882a-2987-41c0-822b-29c21fb5b622\") " Sep 07 13:08:52.470105 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.470081 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f80882a-2987-41c0-822b-29c21fb5b622-kube-api-access-j28hm" (OuterVolumeSpecName: "kube-api-access-j28hm") pod "5f80882a-2987-41c0-822b-29c21fb5b622" (UID: "5f80882a-2987-41c0-822b-29c21fb5b622"). InnerVolumeSpecName "kube-api-access-j28hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:08:52.508263 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.508241 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq" Sep 07 13:08:52.569026 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.568997 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j28hm\" (UniqueName: \"kubernetes.io/projected/5f80882a-2987-41c0-822b-29c21fb5b622-kube-api-access-j28hm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:08:52.792976 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.792946 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq"] Sep 07 13:08:52.794528 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:08:52.794501 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a5e5de_a0ca_45cf_b61d_c82ade4c22f4.slice/crio-fe56230cc1f968f4c31e2180bef985285a28cf271f225f547169f723c5bb031c WatchSource:0}: Error finding container fe56230cc1f968f4c31e2180bef985285a28cf271f225f547169f723c5bb031c: Status 404 returned error can't find the container with id fe56230cc1f968f4c31e2180bef985285a28cf271f225f547169f723c5bb031c Sep 07 13:08:52.825902 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:52.825878 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f80882a-2987-41c0-822b-29c21fb5b622" path="/var/lib/kubelet/pods/5f80882a-2987-41c0-822b-29c21fb5b622/volumes" Sep 07 13:08:53.223403 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:53.223372 2565 generic.go:356] "Generic (PLEG): container finished" podID="5f80882a-2987-41c0-822b-29c21fb5b622" containerID="cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107" exitCode=0 Sep 07 13:08:53.223836 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:53.223415 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fggpc" Sep 07 13:08:53.223836 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:53.223465 2565 scope.go:117] "RemoveContainer" containerID="cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107" Sep 07 13:08:53.225215 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:53.225188 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq" event={"ID":"a6a5e5de-a0ca-45cf-b61d-c82ade4c22f4","Type":"ContainerStarted","Data":"e4532555e4635eb5c2324509b74eba3d8989df80ec3900f6ba57bcf3da49d3a1"} Sep 07 13:08:53.225335 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:53.225242 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq" event={"ID":"a6a5e5de-a0ca-45cf-b61d-c82ade4c22f4","Type":"ContainerStarted","Data":"fe56230cc1f968f4c31e2180bef985285a28cf271f225f547169f723c5bb031c"} Sep 07 13:08:53.225408 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:53.225371 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq" Sep 07 13:08:53.231586 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:53.231570 2565 scope.go:117] "RemoveContainer" containerID="cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107" Sep 07 13:08:53.231831 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:08:53.231816 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107\": container with ID starting with cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107 not found: ID does not exist" containerID="cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107" Sep 07 13:08:53.231879 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:53.231839 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107"} err="failed to get container status \"cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107\": rpc error: code = NotFound desc = could not find container \"cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107\": container with ID starting with cde2cfc1bbc94e42631f83f923ea5dd99ad4388273dc88b8138dbbfb5635d107 not found: ID does not exist" Sep 07 13:08:53.266226 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:08:53.266186 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq" podStartSLOduration=1.266173371 podStartE2EDuration="1.266173371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:08:53.264687833 +0000 UTC m=+554.945073605" watchObservedRunningTime="2026-09-07 13:08:53.266173371 +0000 UTC m=+554.946559141" Sep 07 13:09:04.232612 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:04.232571 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-n42fq" Sep 07 13:09:27.214474 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.214438 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-zwbmf"] Sep 07 13:09:27.221737 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.221721 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-zwbmf" Sep 07 13:09:27.230456 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.230438 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2dhv4\"" Sep 07 13:09:27.252457 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.252430 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-zwbmf"] Sep 07 13:09:27.344991 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.344956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdjc\" (UniqueName: \"kubernetes.io/projected/8bfc631b-538c-405d-a8dd-73101ddd5c24-kube-api-access-7pdjc\") pod \"authorino-b54df94b4-zwbmf\" (UID: \"8bfc631b-538c-405d-a8dd-73101ddd5c24\") " pod="kuadrant-system/authorino-b54df94b4-zwbmf" Sep 07 13:09:27.445427 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.445400 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdjc\" (UniqueName: \"kubernetes.io/projected/8bfc631b-538c-405d-a8dd-73101ddd5c24-kube-api-access-7pdjc\") pod \"authorino-b54df94b4-zwbmf\" (UID: \"8bfc631b-538c-405d-a8dd-73101ddd5c24\") " pod="kuadrant-system/authorino-b54df94b4-zwbmf" Sep 07 13:09:27.461003 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.460972 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdjc\" (UniqueName: \"kubernetes.io/projected/8bfc631b-538c-405d-a8dd-73101ddd5c24-kube-api-access-7pdjc\") pod \"authorino-b54df94b4-zwbmf\" (UID: \"8bfc631b-538c-405d-a8dd-73101ddd5c24\") " pod="kuadrant-system/authorino-b54df94b4-zwbmf" Sep 07 13:09:27.537578 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.537511 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-zwbmf" Sep 07 13:09:27.847527 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.847444 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-zwbmf"] Sep 07 13:09:27.853397 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:09:27.853370 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bfc631b_538c_405d_a8dd_73101ddd5c24.slice/crio-87be426c5897769b62bcc639f3a987372d1f1529c01d49b610f28c052207e130 WatchSource:0}: Error finding container 87be426c5897769b62bcc639f3a987372d1f1529c01d49b610f28c052207e130: Status 404 returned error can't find the container with id 87be426c5897769b62bcc639f3a987372d1f1529c01d49b610f28c052207e130 Sep 07 13:09:27.893792 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.893763 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vvbk"] Sep 07 13:09:27.898497 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.898480 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8vvbk" Sep 07 13:09:27.931078 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.931050 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vvbk"] Sep 07 13:09:27.948677 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:27.948654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sjbq\" (UniqueName: \"kubernetes.io/projected/3c26e43d-38c9-4957-9dfa-3e8883cd08ba-kube-api-access-7sjbq\") pod \"authorino-c548b5d7d-8vvbk\" (UID: \"3c26e43d-38c9-4957-9dfa-3e8883cd08ba\") " pod="kuadrant-system/authorino-c548b5d7d-8vvbk" Sep 07 13:09:28.049385 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:28.049354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sjbq\" (UniqueName: \"kubernetes.io/projected/3c26e43d-38c9-4957-9dfa-3e8883cd08ba-kube-api-access-7sjbq\") pod \"authorino-c548b5d7d-8vvbk\" (UID: \"3c26e43d-38c9-4957-9dfa-3e8883cd08ba\") " pod="kuadrant-system/authorino-c548b5d7d-8vvbk" Sep 07 13:09:28.075826 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:28.075804 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sjbq\" (UniqueName: \"kubernetes.io/projected/3c26e43d-38c9-4957-9dfa-3e8883cd08ba-kube-api-access-7sjbq\") pod \"authorino-c548b5d7d-8vvbk\" (UID: \"3c26e43d-38c9-4957-9dfa-3e8883cd08ba\") " pod="kuadrant-system/authorino-c548b5d7d-8vvbk" Sep 07 13:09:28.207306 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:28.207278 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8vvbk" Sep 07 13:09:28.376390 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:28.376350 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-zwbmf" event={"ID":"8bfc631b-538c-405d-a8dd-73101ddd5c24","Type":"ContainerStarted","Data":"87be426c5897769b62bcc639f3a987372d1f1529c01d49b610f28c052207e130"} Sep 07 13:09:28.496853 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:28.496831 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vvbk"] Sep 07 13:09:28.498495 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:09:28.498467 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c26e43d_38c9_4957_9dfa_3e8883cd08ba.slice/crio-ee242a36f1f61a4281ed0f713494a15ba52fd2879835fb5af81739518e36a2dc WatchSource:0}: Error finding container ee242a36f1f61a4281ed0f713494a15ba52fd2879835fb5af81739518e36a2dc: Status 404 returned error can't find the container with id ee242a36f1f61a4281ed0f713494a15ba52fd2879835fb5af81739518e36a2dc Sep 07 13:09:29.385949 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:29.385857 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8vvbk" event={"ID":"3c26e43d-38c9-4957-9dfa-3e8883cd08ba","Type":"ContainerStarted","Data":"ee242a36f1f61a4281ed0f713494a15ba52fd2879835fb5af81739518e36a2dc"} Sep 07 13:09:31.398120 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:31.398084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-zwbmf" event={"ID":"8bfc631b-538c-405d-a8dd-73101ddd5c24","Type":"ContainerStarted","Data":"744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2"} Sep 07 13:09:31.399818 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:31.399790 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8vvbk" event={"ID":"3c26e43d-38c9-4957-9dfa-3e8883cd08ba","Type":"ContainerStarted","Data":"c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa"} Sep 07 13:09:31.423260 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:31.423218 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-zwbmf" podStartSLOduration=1.783731159 podStartE2EDuration="4.423205038s" totalImagesPullingTime="2.639473879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:09:31.421139222 +0000 UTC m=+593.101524991" watchObservedRunningTime="2026-09-07 13:09:31.423205038 +0000 UTC m=+593.103590808" Sep 07 13:09:31.511841 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:31.511790 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-8vvbk" podStartSLOduration=2.507620207 podStartE2EDuration="4.51177448s" totalImagesPullingTime="2.004154273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:09:31.458011041 +0000 UTC m=+593.138396810" watchObservedRunningTime="2026-09-07 13:09:31.51177448 +0000 UTC m=+593.192160249" Sep 07 13:09:31.512914 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:31.512894 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-zwbmf"] Sep 07 13:09:33.408388 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:33.408331 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-zwbmf" podUID="8bfc631b-538c-405d-a8dd-73101ddd5c24" containerName="authorino" containerID="cri-o://744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2" gracePeriod=30 Sep 07 13:09:33.654944 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:33.654920 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-zwbmf" Sep 07 13:09:33.796811 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:33.796736 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pdjc\" (UniqueName: \"kubernetes.io/projected/8bfc631b-538c-405d-a8dd-73101ddd5c24-kube-api-access-7pdjc\") pod \"8bfc631b-538c-405d-a8dd-73101ddd5c24\" (UID: \"8bfc631b-538c-405d-a8dd-73101ddd5c24\") " Sep 07 13:09:33.798748 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:33.798716 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfc631b-538c-405d-a8dd-73101ddd5c24-kube-api-access-7pdjc" (OuterVolumeSpecName: "kube-api-access-7pdjc") pod "8bfc631b-538c-405d-a8dd-73101ddd5c24" (UID: "8bfc631b-538c-405d-a8dd-73101ddd5c24"). InnerVolumeSpecName "kube-api-access-7pdjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:09:33.898021 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:33.898000 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pdjc\" (UniqueName: \"kubernetes.io/projected/8bfc631b-538c-405d-a8dd-73101ddd5c24-kube-api-access-7pdjc\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:09:34.414142 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:34.414106 2565 generic.go:356] "Generic (PLEG): container finished" podID="8bfc631b-538c-405d-a8dd-73101ddd5c24" containerID="744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2" exitCode=0 Sep 07 13:09:34.414585 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:34.414148 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-zwbmf" Sep 07 13:09:34.414585 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:34.414195 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-zwbmf" event={"ID":"8bfc631b-538c-405d-a8dd-73101ddd5c24","Type":"ContainerDied","Data":"744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2"} Sep 07 13:09:34.414585 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:34.414246 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-zwbmf" event={"ID":"8bfc631b-538c-405d-a8dd-73101ddd5c24","Type":"ContainerDied","Data":"87be426c5897769b62bcc639f3a987372d1f1529c01d49b610f28c052207e130"} Sep 07 13:09:34.414585 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:34.414270 2565 scope.go:117] "RemoveContainer" containerID="744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2" Sep 07 13:09:34.423320 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:34.423302 2565 scope.go:117] "RemoveContainer" containerID="744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2" Sep 07 13:09:34.423619 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:09:34.423588 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2\": container with ID starting with 744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2 not found: ID does not exist" containerID="744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2" Sep 07 13:09:34.423682 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:34.423628 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2"} err="failed to get container status \"744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2\": rpc error: code = NotFound desc = could not find container \"744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2\": container with ID starting with 744a7435858af969a255621dcdc368206d489e672c8f166e1d2ee03c7217e1d2 not found: ID does not exist" Sep 07 13:09:34.439207 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:34.439172 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-zwbmf"] Sep 07 13:09:34.441798 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:34.441779 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-zwbmf"] Sep 07 13:09:34.825832 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:34.825760 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bfc631b-538c-405d-a8dd-73101ddd5c24" path="/var/lib/kubelet/pods/8bfc631b-538c-405d-a8dd-73101ddd5c24/volumes" Sep 07 13:09:55.549800 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.549757 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-b2nmj"] Sep 07 13:09:55.550353 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.550089 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bfc631b-538c-405d-a8dd-73101ddd5c24" containerName="authorino" Sep 07 13:09:55.550353 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.550101 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bfc631b-538c-405d-a8dd-73101ddd5c24" containerName="authorino" Sep 07 13:09:55.550353 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.550170 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bfc631b-538c-405d-a8dd-73101ddd5c24" containerName="authorino" Sep 07 13:09:55.556004 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.555984 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" Sep 07 13:09:55.562892 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.562547 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-b2nmj"] Sep 07 13:09:55.577216 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.577193 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwxg\" (UniqueName: \"kubernetes.io/projected/6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a-kube-api-access-5lwxg\") pod \"authorino-5dc7ccb5f-b2nmj\" (UID: \"6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a\") " pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" Sep 07 13:09:55.678353 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.678317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwxg\" (UniqueName: \"kubernetes.io/projected/6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a-kube-api-access-5lwxg\") pod \"authorino-5dc7ccb5f-b2nmj\" (UID: \"6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a\") " pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" Sep 07 13:09:55.692468 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.692437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwxg\" (UniqueName: \"kubernetes.io/projected/6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a-kube-api-access-5lwxg\") pod \"authorino-5dc7ccb5f-b2nmj\" (UID: \"6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a\") " pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" Sep 07 13:09:55.718361 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.718337 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-b2nmj"] Sep 07 13:09:55.718578 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.718567 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" Sep 07 13:09:55.765943 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.765915 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-9spfq"] Sep 07 13:09:55.770399 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.770373 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-9spfq" Sep 07 13:09:55.774840 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.774811 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 07 13:09:55.779501 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.779343 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5qp\" (UniqueName: \"kubernetes.io/projected/a856c249-1290-4b90-8055-a489cb33ef09-kube-api-access-6f5qp\") pod \"authorino-687bd45585-9spfq\" (UID: \"a856c249-1290-4b90-8055-a489cb33ef09\") " pod="kuadrant-system/authorino-687bd45585-9spfq" Sep 07 13:09:55.779501 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.779419 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a856c249-1290-4b90-8055-a489cb33ef09-tls-cert\") pod \"authorino-687bd45585-9spfq\" (UID: \"a856c249-1290-4b90-8055-a489cb33ef09\") " pod="kuadrant-system/authorino-687bd45585-9spfq" Sep 07 13:09:55.783013 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.782960 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-9spfq"] Sep 07 13:09:55.847386 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.847326 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-9spfq"] Sep 07 13:09:55.847585 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:09:55.847568 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6f5qp tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-9spfq" podUID="a856c249-1290-4b90-8055-a489cb33ef09" Sep 07 13:09:55.880028 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.880000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5qp\" (UniqueName: \"kubernetes.io/projected/a856c249-1290-4b90-8055-a489cb33ef09-kube-api-access-6f5qp\") pod \"authorino-687bd45585-9spfq\" (UID: \"a856c249-1290-4b90-8055-a489cb33ef09\") " pod="kuadrant-system/authorino-687bd45585-9spfq" Sep 07 13:09:55.880131 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.880049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a856c249-1290-4b90-8055-a489cb33ef09-tls-cert\") pod \"authorino-687bd45585-9spfq\" (UID: \"a856c249-1290-4b90-8055-a489cb33ef09\") " pod="kuadrant-system/authorino-687bd45585-9spfq" Sep 07 13:09:55.882302 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.882282 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a856c249-1290-4b90-8055-a489cb33ef09-tls-cert\") pod \"authorino-687bd45585-9spfq\" (UID: \"a856c249-1290-4b90-8055-a489cb33ef09\") " pod="kuadrant-system/authorino-687bd45585-9spfq" Sep 07 13:09:55.896534 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.896505 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5qp\" (UniqueName: \"kubernetes.io/projected/a856c249-1290-4b90-8055-a489cb33ef09-kube-api-access-6f5qp\") pod \"authorino-687bd45585-9spfq\" (UID: \"a856c249-1290-4b90-8055-a489cb33ef09\") " pod="kuadrant-system/authorino-687bd45585-9spfq" Sep 07 13:09:55.904374 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.904352 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-576fd7d96f-8tzk5"] Sep 07 13:09:55.908332 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.908318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-576fd7d96f-8tzk5" Sep 07 13:09:55.934766 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.934745 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-576fd7d96f-8tzk5"] Sep 07 13:09:55.980369 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.980347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4j94\" (UniqueName: \"kubernetes.io/projected/b77fd1fd-4f93-4000-a089-cea797352406-kube-api-access-j4j94\") pod \"authorino-576fd7d96f-8tzk5\" (UID: \"b77fd1fd-4f93-4000-a089-cea797352406\") " pod="kuadrant-system/authorino-576fd7d96f-8tzk5" Sep 07 13:09:55.980477 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.980384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b77fd1fd-4f93-4000-a089-cea797352406-tls-cert\") pod \"authorino-576fd7d96f-8tzk5\" (UID: \"b77fd1fd-4f93-4000-a089-cea797352406\") " pod="kuadrant-system/authorino-576fd7d96f-8tzk5" Sep 07 13:09:55.996927 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:55.996907 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-b2nmj"] Sep 07 13:09:55.999149 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:09:55.999127 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c54554e_7c4a_4c1f_a71a_b21ca0af6c9a.slice/crio-b9180bd141fc5517e3b7e7145a3aba6d1b3ed6ef563976bfbd84f1af28690451 WatchSource:0}: Error finding container b9180bd141fc5517e3b7e7145a3aba6d1b3ed6ef563976bfbd84f1af28690451: Status 404 returned error can't find the container with id b9180bd141fc5517e3b7e7145a3aba6d1b3ed6ef563976bfbd84f1af28690451 Sep 07 13:09:56.080967 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.080936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4j94\" (UniqueName: \"kubernetes.io/projected/b77fd1fd-4f93-4000-a089-cea797352406-kube-api-access-j4j94\") pod \"authorino-576fd7d96f-8tzk5\" (UID: \"b77fd1fd-4f93-4000-a089-cea797352406\") " pod="kuadrant-system/authorino-576fd7d96f-8tzk5" Sep 07 13:09:56.081119 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.080984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b77fd1fd-4f93-4000-a089-cea797352406-tls-cert\") pod \"authorino-576fd7d96f-8tzk5\" (UID: \"b77fd1fd-4f93-4000-a089-cea797352406\") " pod="kuadrant-system/authorino-576fd7d96f-8tzk5" Sep 07 13:09:56.083332 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.083301 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b77fd1fd-4f93-4000-a089-cea797352406-tls-cert\") pod \"authorino-576fd7d96f-8tzk5\" (UID: \"b77fd1fd-4f93-4000-a089-cea797352406\") " pod="kuadrant-system/authorino-576fd7d96f-8tzk5" Sep 07 13:09:56.089871 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.089847 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4j94\" (UniqueName: \"kubernetes.io/projected/b77fd1fd-4f93-4000-a089-cea797352406-kube-api-access-j4j94\") pod \"authorino-576fd7d96f-8tzk5\" (UID: \"b77fd1fd-4f93-4000-a089-cea797352406\") " pod="kuadrant-system/authorino-576fd7d96f-8tzk5" Sep 07 13:09:56.216863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.216844 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-576fd7d96f-8tzk5" Sep 07 13:09:56.502438 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.502357 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-576fd7d96f-8tzk5"] Sep 07 13:09:56.507977 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:09:56.507940 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb77fd1fd_4f93_4000_a089_cea797352406.slice/crio-33578a17af1c60473b4e3dfd4968cad7a4ad5808307dddd044c8c7157c586982 WatchSource:0}: Error finding container 33578a17af1c60473b4e3dfd4968cad7a4ad5808307dddd044c8c7157c586982: Status 404 returned error can't find the container with id 33578a17af1c60473b4e3dfd4968cad7a4ad5808307dddd044c8c7157c586982 Sep 07 13:09:56.514069 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.514041 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-9spfq" Sep 07 13:09:56.514207 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.514068 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" event={"ID":"6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a","Type":"ContainerStarted","Data":"4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88"} Sep 07 13:09:56.514207 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.514092 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" podUID="6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a" containerName="authorino" containerID="cri-o://4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88" gracePeriod=30 Sep 07 13:09:56.514207 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.514111 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" event={"ID":"6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a","Type":"ContainerStarted","Data":"b9180bd141fc5517e3b7e7145a3aba6d1b3ed6ef563976bfbd84f1af28690451"} Sep 07 13:09:56.522237 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.522216 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-9spfq" Sep 07 13:09:56.559363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.559318 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" podStartSLOduration=1.215493612 podStartE2EDuration="1.559301745s" totalImagesPullingTime="343.808133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:09:56.558874292 +0000 UTC m=+618.239260062" watchObservedRunningTime="2026-09-07 13:09:56.559301745 +0000 UTC m=+618.239687515" Sep 07 13:09:56.585064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.585040 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a856c249-1290-4b90-8055-a489cb33ef09-tls-cert\") pod \"a856c249-1290-4b90-8055-a489cb33ef09\" (UID: \"a856c249-1290-4b90-8055-a489cb33ef09\") " Sep 07 13:09:56.585140 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.585090 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f5qp\" (UniqueName: \"kubernetes.io/projected/a856c249-1290-4b90-8055-a489cb33ef09-kube-api-access-6f5qp\") pod \"a856c249-1290-4b90-8055-a489cb33ef09\" (UID: \"a856c249-1290-4b90-8055-a489cb33ef09\") " Sep 07 13:09:56.587083 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.587056 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a856c249-1290-4b90-8055-a489cb33ef09-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a856c249-1290-4b90-8055-a489cb33ef09" (UID: "a856c249-1290-4b90-8055-a489cb33ef09"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:09:56.587162 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.587123 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a856c249-1290-4b90-8055-a489cb33ef09-kube-api-access-6f5qp" (OuterVolumeSpecName: "kube-api-access-6f5qp") pod "a856c249-1290-4b90-8055-a489cb33ef09" (UID: "a856c249-1290-4b90-8055-a489cb33ef09"). InnerVolumeSpecName "kube-api-access-6f5qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:09:56.686232 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.686200 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f5qp\" (UniqueName: \"kubernetes.io/projected/a856c249-1290-4b90-8055-a489cb33ef09-kube-api-access-6f5qp\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:09:56.686355 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.686233 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a856c249-1290-4b90-8055-a489cb33ef09-tls-cert\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:09:56.752098 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.752076 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" Sep 07 13:09:56.787262 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.787210 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lwxg\" (UniqueName: \"kubernetes.io/projected/6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a-kube-api-access-5lwxg\") pod \"6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a\" (UID: \"6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a\") " Sep 07 13:09:56.789176 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.789155 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a-kube-api-access-5lwxg" (OuterVolumeSpecName: "kube-api-access-5lwxg") pod "6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a" (UID: "6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a"). InnerVolumeSpecName "kube-api-access-5lwxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:09:56.888366 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:56.888344 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lwxg\" (UniqueName: \"kubernetes.io/projected/6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a-kube-api-access-5lwxg\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:09:57.520493 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.520459 2565 generic.go:356] "Generic (PLEG): container finished" podID="6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a" containerID="4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88" exitCode=0 Sep 07 13:09:57.520740 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.520504 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" Sep 07 13:09:57.520740 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.520549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" event={"ID":"6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a","Type":"ContainerDied","Data":"4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88"} Sep 07 13:09:57.520740 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.520588 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-b2nmj" event={"ID":"6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a","Type":"ContainerDied","Data":"b9180bd141fc5517e3b7e7145a3aba6d1b3ed6ef563976bfbd84f1af28690451"} Sep 07 13:09:57.520740 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.520632 2565 scope.go:117] "RemoveContainer" containerID="4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88" Sep 07 13:09:57.522165 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.522137 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-576fd7d96f-8tzk5" event={"ID":"b77fd1fd-4f93-4000-a089-cea797352406","Type":"ContainerStarted","Data":"fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa"} Sep 07 13:09:57.522165 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.522157 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-9spfq" Sep 07 13:09:57.522328 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.522182 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-576fd7d96f-8tzk5" event={"ID":"b77fd1fd-4f93-4000-a089-cea797352406","Type":"ContainerStarted","Data":"33578a17af1c60473b4e3dfd4968cad7a4ad5808307dddd044c8c7157c586982"} Sep 07 13:09:57.528763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.528747 2565 scope.go:117] "RemoveContainer" containerID="4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88" Sep 07 13:09:57.529022 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:09:57.529005 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88\": container with ID starting with 4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88 not found: ID does not exist" containerID="4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88" Sep 07 13:09:57.529064 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.529032 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88"} err="failed to get container status \"4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88\": rpc error: code = NotFound desc = could not find container \"4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88\": container with ID starting with 4e2e3b63d84db9fedd182bb636401cedea2ceade9f12ae35c05a58011e1b3f88 not found: ID does not exist" Sep 07 13:09:57.558791 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.558726 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-576fd7d96f-8tzk5" podStartSLOduration=1.987936095 podStartE2EDuration="2.558712011s" totalImagesPullingTime="570.775916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:09:57.556718075 +0000 UTC m=+619.237103845" watchObservedRunningTime="2026-09-07 13:09:57.558712011 +0000 UTC m=+619.239097818" Sep 07 13:09:57.580935 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.580908 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-b2nmj"] Sep 07 13:09:57.586113 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.586089 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-b2nmj"] Sep 07 13:09:57.602377 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.602353 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vvbk"] Sep 07 13:09:57.602563 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.602511 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-8vvbk" podUID="3c26e43d-38c9-4957-9dfa-3e8883cd08ba" containerName="authorino" containerID="cri-o://c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa" gracePeriod=30 Sep 07 13:09:57.637553 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.637524 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-9spfq"] Sep 07 13:09:57.640400 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.640376 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-9spfq"] Sep 07 13:09:57.854160 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.854139 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8vvbk" Sep 07 13:09:57.896709 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.896682 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sjbq\" (UniqueName: \"kubernetes.io/projected/3c26e43d-38c9-4957-9dfa-3e8883cd08ba-kube-api-access-7sjbq\") pod \"3c26e43d-38c9-4957-9dfa-3e8883cd08ba\" (UID: \"3c26e43d-38c9-4957-9dfa-3e8883cd08ba\") " Sep 07 13:09:57.898660 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.898632 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c26e43d-38c9-4957-9dfa-3e8883cd08ba-kube-api-access-7sjbq" (OuterVolumeSpecName: "kube-api-access-7sjbq") pod "3c26e43d-38c9-4957-9dfa-3e8883cd08ba" (UID: "3c26e43d-38c9-4957-9dfa-3e8883cd08ba"). InnerVolumeSpecName "kube-api-access-7sjbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:09:57.997157 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:57.997132 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sjbq\" (UniqueName: \"kubernetes.io/projected/3c26e43d-38c9-4957-9dfa-3e8883cd08ba-kube-api-access-7sjbq\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:09:58.528254 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.528217 2565 generic.go:356] "Generic (PLEG): container finished" podID="3c26e43d-38c9-4957-9dfa-3e8883cd08ba" containerID="c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa" exitCode=0 Sep 07 13:09:58.528420 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.528270 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-8vvbk" Sep 07 13:09:58.528420 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.528305 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8vvbk" event={"ID":"3c26e43d-38c9-4957-9dfa-3e8883cd08ba","Type":"ContainerDied","Data":"c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa"} Sep 07 13:09:58.528420 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.528344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-8vvbk" event={"ID":"3c26e43d-38c9-4957-9dfa-3e8883cd08ba","Type":"ContainerDied","Data":"ee242a36f1f61a4281ed0f713494a15ba52fd2879835fb5af81739518e36a2dc"} Sep 07 13:09:58.528420 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.528367 2565 scope.go:117] "RemoveContainer" containerID="c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa" Sep 07 13:09:58.536799 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.536782 2565 scope.go:117] "RemoveContainer" containerID="c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa" Sep 07 13:09:58.537044 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:09:58.537023 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa\": container with ID starting with c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa not found: ID does not exist" containerID="c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa" Sep 07 13:09:58.537095 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.537051 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa"} err="failed to get container status \"c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa\": rpc error: code = NotFound desc = could not find container \"c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa\": container with ID starting with c711fb515a6ba2f1988c9bcfab74736e9bc87fcac5199ba3414c663e890c0ffa not found: ID does not exist" Sep 07 13:09:58.549399 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.549378 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vvbk"] Sep 07 13:09:58.553831 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.553814 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-8vvbk"] Sep 07 13:09:58.826410 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.826334 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c26e43d-38c9-4957-9dfa-3e8883cd08ba" path="/var/lib/kubelet/pods/3c26e43d-38c9-4957-9dfa-3e8883cd08ba/volumes" Sep 07 13:09:58.826738 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.826658 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a" path="/var/lib/kubelet/pods/6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a/volumes" Sep 07 13:09:58.826971 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:09:58.826957 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a856c249-1290-4b90-8055-a489cb33ef09" path="/var/lib/kubelet/pods/a856c249-1290-4b90-8055-a489cb33ef09/volumes" Sep 07 13:10:42.937803 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:42.937765 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5"] Sep 07 13:10:42.938302 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:42.938216 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c26e43d-38c9-4957-9dfa-3e8883cd08ba" containerName="authorino" Sep 07 13:10:42.938302 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:42.938232 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c26e43d-38c9-4957-9dfa-3e8883cd08ba" containerName="authorino" Sep 07 13:10:42.938302 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:42.938251 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a" containerName="authorino" Sep 07 13:10:42.938302 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:42.938260 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a" containerName="authorino" Sep 07 13:10:42.938521 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:42.938324 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c26e43d-38c9-4957-9dfa-3e8883cd08ba" containerName="authorino" Sep 07 13:10:42.938521 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:42.938339 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c54554e-7c4a-4c1f-a71a-b21ca0af6c9a" containerName="authorino" Sep 07 13:10:42.942761 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:42.942739 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" Sep 07 13:10:42.946758 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:42.946735 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cxmkd\"" Sep 07 13:10:42.946922 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:42.946863 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 07 13:10:42.987279 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:42.987241 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5"] Sep 07 13:10:43.088680 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:43.088646 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6009be81-7b8d-44b4-8fa4-d8c3da6736cc-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6bld5\" (UID: \"6009be81-7b8d-44b4-8fa4-d8c3da6736cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" Sep 07 13:10:43.088851 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:43.088690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdqc5\" (UniqueName: \"kubernetes.io/projected/6009be81-7b8d-44b4-8fa4-d8c3da6736cc-kube-api-access-vdqc5\") pod \"payload-pre-processing-59fbc47bf8-6bld5\" (UID: \"6009be81-7b8d-44b4-8fa4-d8c3da6736cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" Sep 07 13:10:43.189936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:43.189849 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6009be81-7b8d-44b4-8fa4-d8c3da6736cc-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6bld5\" (UID: \"6009be81-7b8d-44b4-8fa4-d8c3da6736cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" Sep 07 13:10:43.189936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:43.189896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqc5\" (UniqueName: \"kubernetes.io/projected/6009be81-7b8d-44b4-8fa4-d8c3da6736cc-kube-api-access-vdqc5\") pod \"payload-pre-processing-59fbc47bf8-6bld5\" (UID: \"6009be81-7b8d-44b4-8fa4-d8c3da6736cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" Sep 07 13:10:43.190620 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:43.190574 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6009be81-7b8d-44b4-8fa4-d8c3da6736cc-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6bld5\" (UID: \"6009be81-7b8d-44b4-8fa4-d8c3da6736cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" Sep 07 13:10:43.203333 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:43.203310 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdqc5\" (UniqueName: \"kubernetes.io/projected/6009be81-7b8d-44b4-8fa4-d8c3da6736cc-kube-api-access-vdqc5\") pod \"payload-pre-processing-59fbc47bf8-6bld5\" (UID: \"6009be81-7b8d-44b4-8fa4-d8c3da6736cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" Sep 07 13:10:43.253327 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:43.253290 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" Sep 07 13:10:43.537766 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:43.537700 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5"] Sep 07 13:10:43.540213 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:10:43.540184 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6009be81_7b8d_44b4_8fa4_d8c3da6736cc.slice/crio-4afeab99f1df1e8c5784355c23faaddfa2938ec803366cf39ad24c4dd71e4b55 WatchSource:0}: Error finding container 4afeab99f1df1e8c5784355c23faaddfa2938ec803366cf39ad24c4dd71e4b55: Status 404 returned error can't find the container with id 4afeab99f1df1e8c5784355c23faaddfa2938ec803366cf39ad24c4dd71e4b55 Sep 07 13:10:43.739684 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:43.739649 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" event={"ID":"6009be81-7b8d-44b4-8fa4-d8c3da6736cc","Type":"ContainerStarted","Data":"4afeab99f1df1e8c5784355c23faaddfa2938ec803366cf39ad24c4dd71e4b55"} Sep 07 13:10:46.755643 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:46.755545 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" event={"ID":"6009be81-7b8d-44b4-8fa4-d8c3da6736cc","Type":"ContainerStarted","Data":"9b8f7d667c7cb74012bce7d376b04fb0272499b77c27de786d6deb243bfd2b04"} Sep 07 13:10:46.755992 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:46.755674 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" Sep 07 13:10:46.777046 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:46.776998 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" podStartSLOduration=1.874787486 podStartE2EDuration="4.776982505s" totalImagesPullingTime="2.902195019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:10:46.774629761 +0000 UTC m=+668.455015525" watchObservedRunningTime="2026-09-07 13:10:46.776982505 +0000 UTC m=+668.457368314" Sep 07 13:10:57.762256 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:10:57.762223 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6bld5" Sep 07 13:11:23.558969 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.558938 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch"] Sep 07 13:11:23.565757 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.565731 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.569952 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.569933 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 07 13:11:23.570339 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.570319 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 07 13:11:23.570757 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.570741 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-45cgz\"" Sep 07 13:11:23.570840 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.570776 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 07 13:11:23.589082 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.589055 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch"] Sep 07 13:11:23.642252 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.642220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46af2c4-393c-485d-84ef-53e01d008442-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.642252 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.642256 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.642463 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.642285 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.642463 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.642329 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.642463 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.642367 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twn7v\" (UniqueName: \"kubernetes.io/projected/a46af2c4-393c-485d-84ef-53e01d008442-kube-api-access-twn7v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.642463 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.642395 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.743510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.743461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.743700 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.743539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46af2c4-393c-485d-84ef-53e01d008442-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.743700 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.743558 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.743700 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.743588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.743700 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.743633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.743700 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.743661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twn7v\" (UniqueName: \"kubernetes.io/projected/a46af2c4-393c-485d-84ef-53e01d008442-kube-api-access-twn7v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.744014 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.743978 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.744067 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.744013 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.744067 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.744048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.745989 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.745966 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46af2c4-393c-485d-84ef-53e01d008442-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.746101 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.746087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46af2c4-393c-485d-84ef-53e01d008442-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.757495 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.757470 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twn7v\" (UniqueName: \"kubernetes.io/projected/a46af2c4-393c-485d-84ef-53e01d008442-kube-api-access-twn7v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch\" (UID: \"a46af2c4-393c-485d-84ef-53e01d008442\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:23.875553 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:23.875470 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:24.168322 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:24.168294 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch"] Sep 07 13:11:24.171000 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:11:24.170975 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46af2c4_393c_485d_84ef_53e01d008442.slice/crio-94d3e45b692f4e3d6d1e25d81c122f7b4760b469296742e379617425ec787207 WatchSource:0}: Error finding container 94d3e45b692f4e3d6d1e25d81c122f7b4760b469296742e379617425ec787207: Status 404 returned error can't find the container with id 94d3e45b692f4e3d6d1e25d81c122f7b4760b469296742e379617425ec787207 Sep 07 13:11:24.172301 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:24.172284 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 13:11:24.926440 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:24.926389 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" event={"ID":"a46af2c4-393c-485d-84ef-53e01d008442","Type":"ContainerStarted","Data":"94d3e45b692f4e3d6d1e25d81c122f7b4760b469296742e379617425ec787207"} Sep 07 13:11:25.932827 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:25.932794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" event={"ID":"a46af2c4-393c-485d-84ef-53e01d008442","Type":"ContainerStarted","Data":"c035101934f192f9df8a6ebfb677c6e2be632da5bbe29eb0fa93676827dce719"} Sep 07 13:11:25.979446 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:25.979397 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" podStartSLOduration=1.870655001 podStartE2EDuration="2.979380843s" totalImagesPullingTime="1.108725842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:11:25.975466549 +0000 UTC m=+707.655852320" watchObservedRunningTime="2026-09-07 13:11:25.979380843 +0000 UTC m=+707.659766612" Sep 07 13:11:27.139216 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.139183 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz"] Sep 07 13:11:27.142774 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.142756 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.149010 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.148989 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 07 13:11:27.206767 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.206734 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz"] Sep 07 13:11:27.273758 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.273710 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx2cd\" (UniqueName: \"kubernetes.io/projected/9ad73588-dd84-4766-9621-4bcfea48abe4-kube-api-access-jx2cd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.273906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.273800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.273906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.273851 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.273906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.273889 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad73588-dd84-4766-9621-4bcfea48abe4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.274029 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.273916 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.274029 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.273956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.374694 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.374647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.374694 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.374700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.374944 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.374727 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad73588-dd84-4766-9621-4bcfea48abe4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.374944 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.374747 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.374944 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.374767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.374944 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.374809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2cd\" (UniqueName: \"kubernetes.io/projected/9ad73588-dd84-4766-9621-4bcfea48abe4-kube-api-access-jx2cd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.375162 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.375063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.375162 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.375101 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.375162 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.375138 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.377117 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.377088 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad73588-dd84-4766-9621-4bcfea48abe4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.377277 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.377258 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad73588-dd84-4766-9621-4bcfea48abe4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.396168 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.396109 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2cd\" (UniqueName: \"kubernetes.io/projected/9ad73588-dd84-4766-9621-4bcfea48abe4-kube-api-access-jx2cd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmwvz\" (UID: \"9ad73588-dd84-4766-9621-4bcfea48abe4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.453023 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.452990 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:27.736261 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.736214 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz"] Sep 07 13:11:27.739922 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:11:27.739896 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad73588_dd84_4766_9621_4bcfea48abe4.slice/crio-f2567acee75367144adf67e94d4df722360603985aa7c523d83502ee89da120c WatchSource:0}: Error finding container f2567acee75367144adf67e94d4df722360603985aa7c523d83502ee89da120c: Status 404 returned error can't find the container with id f2567acee75367144adf67e94d4df722360603985aa7c523d83502ee89da120c Sep 07 13:11:27.943655 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:27.943630 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" event={"ID":"9ad73588-dd84-4766-9621-4bcfea48abe4","Type":"ContainerStarted","Data":"f2567acee75367144adf67e94d4df722360603985aa7c523d83502ee89da120c"} Sep 07 13:11:28.951302 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:28.951258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" event={"ID":"9ad73588-dd84-4766-9621-4bcfea48abe4","Type":"ContainerStarted","Data":"92853c3f8fa181c16f209ce91274f5736e4d6840d4a79e2fc75ed020aa375356"} Sep 07 13:11:28.982888 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:28.982836 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" podStartSLOduration=1.7855539999999999 podStartE2EDuration="1.982819223s" totalImagesPullingTime="197.265223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:11:28.981434394 +0000 UTC m=+710.661820164" watchObservedRunningTime="2026-09-07 13:11:28.982819223 +0000 UTC m=+710.663205025" Sep 07 13:11:33.876290 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:33.876198 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:33.876290 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:33.876270 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:33.888903 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:33.888876 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:33.985651 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:33.985619 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch" Sep 07 13:11:37.453320 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:37.453277 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:37.453720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:37.453353 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:37.466082 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:37.466061 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:38.003667 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:38.003638 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmwvz" Sep 07 13:11:59.599273 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.599239 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg"] Sep 07 13:11:59.603153 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.603135 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.607170 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.607149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 07 13:11:59.641859 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.641827 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg"] Sep 07 13:11:59.754718 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.754685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.754907 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.754727 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dwgz\" (UniqueName: \"kubernetes.io/projected/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-kube-api-access-9dwgz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.754907 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.754747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.754907 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.754849 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.755015 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.754912 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.755015 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.754940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.855717 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.855643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.855717 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.855699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.855933 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.855724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.855933 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.855768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.855933 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.855907 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dwgz\" (UniqueName: \"kubernetes.io/projected/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-kube-api-access-9dwgz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.856079 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.855949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.856227 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.856205 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.856312 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.856274 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.856404 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.856383 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.858033 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.858009 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.858260 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.858243 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.869995 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.869969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dwgz\" (UniqueName: \"kubernetes.io/projected/b3e3c31e-5117-41a8-8e12-a3dcf555f2f7-kube-api-access-9dwgz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg\" (UID: \"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:11:59.911902 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:11:59.911869 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:12:00.195559 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:12:00.195524 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg"] Sep 07 13:12:00.222587 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:12:00.222564 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3e3c31e_5117_41a8_8e12_a3dcf555f2f7.slice/crio-42a2d4a811c35bfb690ef15624080dff5ba08f7be5d38ac131144d44a46e2845 WatchSource:0}: Error finding container 42a2d4a811c35bfb690ef15624080dff5ba08f7be5d38ac131144d44a46e2845: Status 404 returned error can't find the container with id 42a2d4a811c35bfb690ef15624080dff5ba08f7be5d38ac131144d44a46e2845 Sep 07 13:12:01.100603 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:12:01.100571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" event={"ID":"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7","Type":"ContainerStarted","Data":"36693ceac768c793a27ab4cea9b8321cd972c8e19a7ce6046cb20dd3cec5ebdb"} Sep 07 13:12:01.100962 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:12:01.100619 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" event={"ID":"b3e3c31e-5117-41a8-8e12-a3dcf555f2f7","Type":"ContainerStarted","Data":"42a2d4a811c35bfb690ef15624080dff5ba08f7be5d38ac131144d44a46e2845"} Sep 07 13:12:01.163517 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:12:01.163473 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" podStartSLOduration=1.987950298 podStartE2EDuration="2.163459522s" totalImagesPullingTime="175.509224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:12:01.154009962 +0000 UTC m=+742.834395731" watchObservedRunningTime="2026-09-07 13:12:01.163459522 +0000 UTC m=+742.843845292" Sep 07 13:12:09.912124 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:12:09.912089 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:12:09.912493 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:12:09.912137 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:12:09.929151 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:12:09.929126 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:12:10.158229 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:12:10.158196 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg" Sep 07 13:13:34.807169 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:34.807135 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6598d66d5b-6rhbr"] Sep 07 13:13:34.810523 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:34.810504 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6598d66d5b-6rhbr" Sep 07 13:13:34.829448 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:34.829422 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6598d66d5b-6rhbr"] Sep 07 13:13:34.892168 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:34.892138 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/698d6a55-c084-40aa-8358-f0149a0fc6ad-tls-cert\") pod \"authorino-6598d66d5b-6rhbr\" (UID: \"698d6a55-c084-40aa-8358-f0149a0fc6ad\") " pod="kuadrant-system/authorino-6598d66d5b-6rhbr" Sep 07 13:13:34.892317 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:34.892211 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn8v5\" (UniqueName: \"kubernetes.io/projected/698d6a55-c084-40aa-8358-f0149a0fc6ad-kube-api-access-wn8v5\") pod \"authorino-6598d66d5b-6rhbr\" (UID: \"698d6a55-c084-40aa-8358-f0149a0fc6ad\") " pod="kuadrant-system/authorino-6598d66d5b-6rhbr" Sep 07 13:13:34.993449 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:34.993422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn8v5\" (UniqueName: \"kubernetes.io/projected/698d6a55-c084-40aa-8358-f0149a0fc6ad-kube-api-access-wn8v5\") pod \"authorino-6598d66d5b-6rhbr\" (UID: \"698d6a55-c084-40aa-8358-f0149a0fc6ad\") " pod="kuadrant-system/authorino-6598d66d5b-6rhbr" Sep 07 13:13:34.993589 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:34.993502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/698d6a55-c084-40aa-8358-f0149a0fc6ad-tls-cert\") pod \"authorino-6598d66d5b-6rhbr\" (UID: \"698d6a55-c084-40aa-8358-f0149a0fc6ad\") " pod="kuadrant-system/authorino-6598d66d5b-6rhbr" Sep 07 13:13:34.995675 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:34.995655 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/698d6a55-c084-40aa-8358-f0149a0fc6ad-tls-cert\") pod \"authorino-6598d66d5b-6rhbr\" (UID: \"698d6a55-c084-40aa-8358-f0149a0fc6ad\") " pod="kuadrant-system/authorino-6598d66d5b-6rhbr" Sep 07 13:13:35.003211 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:35.003190 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn8v5\" (UniqueName: \"kubernetes.io/projected/698d6a55-c084-40aa-8358-f0149a0fc6ad-kube-api-access-wn8v5\") pod \"authorino-6598d66d5b-6rhbr\" (UID: \"698d6a55-c084-40aa-8358-f0149a0fc6ad\") " pod="kuadrant-system/authorino-6598d66d5b-6rhbr" Sep 07 13:13:35.120862 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:35.120785 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6598d66d5b-6rhbr" Sep 07 13:13:35.390045 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:35.390021 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6598d66d5b-6rhbr"] Sep 07 13:13:35.392237 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:13:35.392201 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod698d6a55_c084_40aa_8358_f0149a0fc6ad.slice/crio-f4115ba8b51b2dc4957fb1297af0b2f8f59430ad43f3bf10b61eb594e7e48a11 WatchSource:0}: Error finding container f4115ba8b51b2dc4957fb1297af0b2f8f59430ad43f3bf10b61eb594e7e48a11: Status 404 returned error can't find the container with id f4115ba8b51b2dc4957fb1297af0b2f8f59430ad43f3bf10b61eb594e7e48a11 Sep 07 13:13:35.535151 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:35.535119 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6598d66d5b-6rhbr" event={"ID":"698d6a55-c084-40aa-8358-f0149a0fc6ad","Type":"ContainerStarted","Data":"f4115ba8b51b2dc4957fb1297af0b2f8f59430ad43f3bf10b61eb594e7e48a11"} Sep 07 13:13:36.542348 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:36.542251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6598d66d5b-6rhbr" event={"ID":"698d6a55-c084-40aa-8358-f0149a0fc6ad","Type":"ContainerStarted","Data":"5a2ee685b1d8dbf377968524890c3b9cd8cdff2a5f44f59b851cb454dad8bb47"} Sep 07 13:13:36.571013 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:36.570959 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6598d66d5b-6rhbr" podStartSLOduration=1.728679574 podStartE2EDuration="2.570941297s" totalImagesPullingTime="842.261723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:13:36.568609747 +0000 UTC m=+838.248995517" watchObservedRunningTime="2026-09-07 13:13:36.570941297 +0000 UTC m=+838.251327066" Sep 07 13:13:36.635718 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:36.635683 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-576fd7d96f-8tzk5"] Sep 07 13:13:36.635949 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:36.635915 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-576fd7d96f-8tzk5" podUID="b77fd1fd-4f93-4000-a089-cea797352406" containerName="authorino" containerID="cri-o://fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa" gracePeriod=30 Sep 07 13:13:36.890100 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:36.890076 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-576fd7d96f-8tzk5" Sep 07 13:13:37.014565 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.014534 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b77fd1fd-4f93-4000-a089-cea797352406-tls-cert\") pod \"b77fd1fd-4f93-4000-a089-cea797352406\" (UID: \"b77fd1fd-4f93-4000-a089-cea797352406\") " Sep 07 13:13:37.014734 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.014590 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4j94\" (UniqueName: \"kubernetes.io/projected/b77fd1fd-4f93-4000-a089-cea797352406-kube-api-access-j4j94\") pod \"b77fd1fd-4f93-4000-a089-cea797352406\" (UID: \"b77fd1fd-4f93-4000-a089-cea797352406\") " Sep 07 13:13:37.016560 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.016531 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77fd1fd-4f93-4000-a089-cea797352406-kube-api-access-j4j94" (OuterVolumeSpecName: "kube-api-access-j4j94") pod "b77fd1fd-4f93-4000-a089-cea797352406" (UID: "b77fd1fd-4f93-4000-a089-cea797352406"). InnerVolumeSpecName "kube-api-access-j4j94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:13:37.025085 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.025053 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b77fd1fd-4f93-4000-a089-cea797352406-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b77fd1fd-4f93-4000-a089-cea797352406" (UID: "b77fd1fd-4f93-4000-a089-cea797352406"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:13:37.116140 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.116069 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b77fd1fd-4f93-4000-a089-cea797352406-tls-cert\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:13:37.116140 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.116100 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4j94\" (UniqueName: \"kubernetes.io/projected/b77fd1fd-4f93-4000-a089-cea797352406-kube-api-access-j4j94\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:13:37.548543 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.548511 2565 generic.go:356] "Generic (PLEG): container finished" podID="b77fd1fd-4f93-4000-a089-cea797352406" containerID="fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa" exitCode=0 Sep 07 13:13:37.548972 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.548582 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-576fd7d96f-8tzk5" Sep 07 13:13:37.548972 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.548629 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-576fd7d96f-8tzk5" event={"ID":"b77fd1fd-4f93-4000-a089-cea797352406","Type":"ContainerDied","Data":"fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa"} Sep 07 13:13:37.548972 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.548672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-576fd7d96f-8tzk5" event={"ID":"b77fd1fd-4f93-4000-a089-cea797352406","Type":"ContainerDied","Data":"33578a17af1c60473b4e3dfd4968cad7a4ad5808307dddd044c8c7157c586982"} Sep 07 13:13:37.548972 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.548694 2565 scope.go:117] "RemoveContainer" containerID="fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa" Sep 07 13:13:37.558901 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.558880 2565 scope.go:117] "RemoveContainer" containerID="fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa" Sep 07 13:13:37.559169 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:13:37.559151 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa\": container with ID starting with fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa not found: ID does not exist" containerID="fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa" Sep 07 13:13:37.559246 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.559181 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa"} err="failed to get container status \"fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa\": rpc error: code = NotFound desc = could not find container \"fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa\": container with ID starting with fc8e93e10e9bac9ed4054714a32729277d2a549fbf446149bf96a56242896faa not found: ID does not exist" Sep 07 13:13:37.571295 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.571270 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-576fd7d96f-8tzk5"] Sep 07 13:13:37.573645 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:37.573622 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-576fd7d96f-8tzk5"] Sep 07 13:13:38.831230 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:13:38.831187 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b77fd1fd-4f93-4000-a089-cea797352406" path="/var/lib/kubelet/pods/b77fd1fd-4f93-4000-a089-cea797352406/volumes" Sep 07 13:14:01.248409 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.248375 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg"] Sep 07 13:14:01.248869 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.248762 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b77fd1fd-4f93-4000-a089-cea797352406" containerName="authorino" Sep 07 13:14:01.248869 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.248774 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b77fd1fd-4f93-4000-a089-cea797352406" containerName="authorino" Sep 07 13:14:01.248869 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.248826 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b77fd1fd-4f93-4000-a089-cea797352406" containerName="authorino" Sep 07 13:14:01.253708 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.253690 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:14:01.256246 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.256225 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ebfa46-dockercfg-bgf47\"" Sep 07 13:14:01.256343 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.256276 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ebfa46\"" Sep 07 13:14:01.265236 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.265214 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg"] Sep 07 13:14:01.294969 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.294947 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d"] Sep 07 13:14:01.298426 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.298410 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:14:01.314108 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.314086 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d"] Sep 07 13:14:01.340396 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.340367 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghzv\" (UniqueName: \"kubernetes.io/projected/798fc516-4df3-40a7-b2dc-473d7de484ef-kube-api-access-nghzv\") pod \"payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d\" (UID: \"798fc516-4df3-40a7-b2dc-473d7de484ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:14:01.340491 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.340437 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvth\" (UniqueName: \"kubernetes.io/projected/3b54bf30-a16b-48b0-b83d-a6722ca30880-kube-api-access-6lvth\") pod \"payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg\" (UID: \"3b54bf30-a16b-48b0-b83d-a6722ca30880\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:14:01.340491 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.340470 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798fc516-4df3-40a7-b2dc-473d7de484ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d\" (UID: \"798fc516-4df3-40a7-b2dc-473d7de484ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:14:01.340623 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.340589 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b54bf30-a16b-48b0-b83d-a6722ca30880-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg\" (UID: \"3b54bf30-a16b-48b0-b83d-a6722ca30880\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:14:01.441816 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.441779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b54bf30-a16b-48b0-b83d-a6722ca30880-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg\" (UID: \"3b54bf30-a16b-48b0-b83d-a6722ca30880\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:14:01.442000 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.441846 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nghzv\" (UniqueName: \"kubernetes.io/projected/798fc516-4df3-40a7-b2dc-473d7de484ef-kube-api-access-nghzv\") pod \"payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d\" (UID: \"798fc516-4df3-40a7-b2dc-473d7de484ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:14:01.442000 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.441883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvth\" (UniqueName: \"kubernetes.io/projected/3b54bf30-a16b-48b0-b83d-a6722ca30880-kube-api-access-6lvth\") pod \"payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg\" (UID: \"3b54bf30-a16b-48b0-b83d-a6722ca30880\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:14:01.442000 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.441912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798fc516-4df3-40a7-b2dc-473d7de484ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d\" (UID: \"798fc516-4df3-40a7-b2dc-473d7de484ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:14:01.442499 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.442474 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b54bf30-a16b-48b0-b83d-a6722ca30880-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg\" (UID: \"3b54bf30-a16b-48b0-b83d-a6722ca30880\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:14:01.442662 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.442642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798fc516-4df3-40a7-b2dc-473d7de484ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d\" (UID: \"798fc516-4df3-40a7-b2dc-473d7de484ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:14:01.452585 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.452561 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvth\" (UniqueName: \"kubernetes.io/projected/3b54bf30-a16b-48b0-b83d-a6722ca30880-kube-api-access-6lvth\") pod \"payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg\" (UID: \"3b54bf30-a16b-48b0-b83d-a6722ca30880\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:14:01.452815 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.452795 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghzv\" (UniqueName: \"kubernetes.io/projected/798fc516-4df3-40a7-b2dc-473d7de484ef-kube-api-access-nghzv\") pod \"payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d\" (UID: \"798fc516-4df3-40a7-b2dc-473d7de484ef\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:14:01.564114 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.564028 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:14:01.607688 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.607638 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:14:01.854283 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.854207 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg"] Sep 07 13:14:01.857324 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:14:01.857301 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b54bf30_a16b_48b0_b83d_a6722ca30880.slice/crio-d149bfb2db7887f5c813bd74e3f173423204bdfad212907434480071c6ce7022 WatchSource:0}: Error finding container d149bfb2db7887f5c813bd74e3f173423204bdfad212907434480071c6ce7022: Status 404 returned error can't find the container with id d149bfb2db7887f5c813bd74e3f173423204bdfad212907434480071c6ce7022 Sep 07 13:14:01.910438 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:01.910412 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d"] Sep 07 13:14:01.912738 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:14:01.912713 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798fc516_4df3_40a7_b2dc_473d7de484ef.slice/crio-2af95d96a61acce376abf35cfe88c98d75de04ba31a4ffb718b103a9d78abcda WatchSource:0}: Error finding container 2af95d96a61acce376abf35cfe88c98d75de04ba31a4ffb718b103a9d78abcda: Status 404 returned error can't find the container with id 2af95d96a61acce376abf35cfe88c98d75de04ba31a4ffb718b103a9d78abcda Sep 07 13:14:02.667019 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:02.666962 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" event={"ID":"3b54bf30-a16b-48b0-b83d-a6722ca30880","Type":"ContainerStarted","Data":"51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6"} Sep 07 13:14:02.667481 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:02.667082 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" event={"ID":"3b54bf30-a16b-48b0-b83d-a6722ca30880","Type":"ContainerStarted","Data":"d149bfb2db7887f5c813bd74e3f173423204bdfad212907434480071c6ce7022"} Sep 07 13:14:02.667481 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:02.667112 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:14:02.668474 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:02.668444 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" event={"ID":"798fc516-4df3-40a7-b2dc-473d7de484ef","Type":"ContainerStarted","Data":"c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539"} Sep 07 13:14:02.668616 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:02.668486 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" event={"ID":"798fc516-4df3-40a7-b2dc-473d7de484ef","Type":"ContainerStarted","Data":"2af95d96a61acce376abf35cfe88c98d75de04ba31a4ffb718b103a9d78abcda"} Sep 07 13:14:02.668616 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:02.668506 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:14:02.690343 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:02.690294 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" podStartSLOduration=1.69028022 podStartE2EDuration="1.69028022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:02.688716615 +0000 UTC m=+864.369102385" watchObservedRunningTime="2026-09-07 13:14:02.69028022 +0000 UTC m=+864.370666026" Sep 07 13:14:02.711492 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:02.711451 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" podStartSLOduration=1.711436991 podStartE2EDuration="1.711436991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:02.711110158 +0000 UTC m=+864.391495929" watchObservedRunningTime="2026-09-07 13:14:02.711436991 +0000 UTC m=+864.391822760" Sep 07 13:14:10.430559 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.430526 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b"] Sep 07 13:14:10.434068 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.434048 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:10.437134 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.437114 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e07a5db5-dockercfg-kc7cp\"" Sep 07 13:14:10.447588 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.447571 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e07a5db5\"" Sep 07 13:14:10.477726 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.477702 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b"] Sep 07 13:14:10.529730 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.529707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwj7\" (UniqueName: \"kubernetes.io/projected/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-kube-api-access-6dwj7\") pod \"payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b\" (UID: \"8909b3c6-7932-492d-890c-0dc6ddc3f0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:10.529856 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.529769 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b\" (UID: \"8909b3c6-7932-492d-890c-0dc6ddc3f0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:10.630559 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.630529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwj7\" (UniqueName: \"kubernetes.io/projected/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-kube-api-access-6dwj7\") pod \"payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b\" (UID: \"8909b3c6-7932-492d-890c-0dc6ddc3f0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:10.630704 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.630573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b\" (UID: \"8909b3c6-7932-492d-890c-0dc6ddc3f0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:10.631205 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.631186 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b\" (UID: \"8909b3c6-7932-492d-890c-0dc6ddc3f0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:10.640638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.640613 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwj7\" (UniqueName: \"kubernetes.io/projected/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-kube-api-access-6dwj7\") pod \"payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b\" (UID: \"8909b3c6-7932-492d-890c-0dc6ddc3f0ba\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:10.694816 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.694757 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq"] Sep 07 13:14:10.698246 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.698230 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:10.707671 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.707646 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq"] Sep 07 13:14:10.744639 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.744613 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:10.832190 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.832155 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-plugins-config-volume\") pod \"payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq\" (UID: \"b0ece7cf-c5e3-421c-9a80-43f8ee060e69\") " pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:10.832372 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.832279 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp96p\" (UniqueName: \"kubernetes.io/projected/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-kube-api-access-lp96p\") pod \"payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq\" (UID: \"b0ece7cf-c5e3-421c-9a80-43f8ee060e69\") " pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:10.933664 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.933626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-plugins-config-volume\") pod \"payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq\" (UID: \"b0ece7cf-c5e3-421c-9a80-43f8ee060e69\") " pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:10.933829 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.933716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp96p\" (UniqueName: \"kubernetes.io/projected/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-kube-api-access-lp96p\") pod \"payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq\" (UID: \"b0ece7cf-c5e3-421c-9a80-43f8ee060e69\") " pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:10.934360 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.934340 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-plugins-config-volume\") pod \"payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq\" (UID: \"b0ece7cf-c5e3-421c-9a80-43f8ee060e69\") " pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:10.942617 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:10.942570 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp96p\" (UniqueName: \"kubernetes.io/projected/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-kube-api-access-lp96p\") pod \"payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq\" (UID: \"b0ece7cf-c5e3-421c-9a80-43f8ee060e69\") " pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:11.009984 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.009907 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:11.032041 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.031979 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b"] Sep 07 13:14:11.034353 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:14:11.034325 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8909b3c6_7932_492d_890c_0dc6ddc3f0ba.slice/crio-aed921dbb2c8ab8753638a10117c2fc397b6005347b2830f2fca09f6b527c11b WatchSource:0}: Error finding container aed921dbb2c8ab8753638a10117c2fc397b6005347b2830f2fca09f6b527c11b: Status 404 returned error can't find the container with id aed921dbb2c8ab8753638a10117c2fc397b6005347b2830f2fca09f6b527c11b Sep 07 13:14:11.289103 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.289080 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq"] Sep 07 13:14:11.291354 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:14:11.291328 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ece7cf_c5e3_421c_9a80_43f8ee060e69.slice/crio-0f42e8e31e48bb12d5cff047ea06a1b2388debfe1759a4ad8bfb90b12de4047d WatchSource:0}: Error finding container 0f42e8e31e48bb12d5cff047ea06a1b2388debfe1759a4ad8bfb90b12de4047d: Status 404 returned error can't find the container with id 0f42e8e31e48bb12d5cff047ea06a1b2388debfe1759a4ad8bfb90b12de4047d Sep 07 13:14:11.714360 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.714328 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" event={"ID":"8909b3c6-7932-492d-890c-0dc6ddc3f0ba","Type":"ContainerStarted","Data":"abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57"} Sep 07 13:14:11.714828 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.714386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" event={"ID":"8909b3c6-7932-492d-890c-0dc6ddc3f0ba","Type":"ContainerStarted","Data":"aed921dbb2c8ab8753638a10117c2fc397b6005347b2830f2fca09f6b527c11b"} Sep 07 13:14:11.714828 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.714440 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:11.716078 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.716055 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" event={"ID":"b0ece7cf-c5e3-421c-9a80-43f8ee060e69","Type":"ContainerStarted","Data":"4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125"} Sep 07 13:14:11.716198 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.716090 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" event={"ID":"b0ece7cf-c5e3-421c-9a80-43f8ee060e69","Type":"ContainerStarted","Data":"0f42e8e31e48bb12d5cff047ea06a1b2388debfe1759a4ad8bfb90b12de4047d"} Sep 07 13:14:11.716198 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.716155 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:11.732882 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.732863 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf"] Sep 07 13:14:11.736507 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.736486 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:14:11.736849 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.736805 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" podStartSLOduration=1.736791151 podStartE2EDuration="1.736791151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:11.735930186 +0000 UTC m=+873.416315959" watchObservedRunningTime="2026-09-07 13:14:11.736791151 +0000 UTC m=+873.417176922" Sep 07 13:14:11.738735 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.738714 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-adacb1-dockercfg-vjx9j\"" Sep 07 13:14:11.738829 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.738714 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-adacb1\"" Sep 07 13:14:11.749096 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.749073 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf"] Sep 07 13:14:11.755173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.755129 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" podStartSLOduration=1.7551133330000002 podStartE2EDuration="1.755113333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:11.754579281 +0000 UTC m=+873.434965053" watchObservedRunningTime="2026-09-07 13:14:11.755113333 +0000 UTC m=+873.435499115" Sep 07 13:14:11.839747 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.839715 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-plugins-config-volume\") pod \"payload-processing-e2e-autores-adacb1-6468f48965-rfbwf\" (UID: \"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a\") " pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:14:11.839900 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.839805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnckk\" (UniqueName: \"kubernetes.io/projected/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-kube-api-access-wnckk\") pod \"payload-processing-e2e-autores-adacb1-6468f48965-rfbwf\" (UID: \"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a\") " pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:14:11.940978 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.940945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-plugins-config-volume\") pod \"payload-processing-e2e-autores-adacb1-6468f48965-rfbwf\" (UID: \"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a\") " pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:14:11.941143 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.941005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnckk\" (UniqueName: \"kubernetes.io/projected/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-kube-api-access-wnckk\") pod \"payload-processing-e2e-autores-adacb1-6468f48965-rfbwf\" (UID: \"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a\") " pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:14:11.941704 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.941680 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-plugins-config-volume\") pod \"payload-processing-e2e-autores-adacb1-6468f48965-rfbwf\" (UID: \"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a\") " pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:14:11.951672 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:11.951651 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnckk\" (UniqueName: \"kubernetes.io/projected/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-kube-api-access-wnckk\") pod \"payload-processing-e2e-autores-adacb1-6468f48965-rfbwf\" (UID: \"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a\") " pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:14:12.048715 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:12.048635 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:14:12.319286 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:12.319261 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf"] Sep 07 13:14:12.321175 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:14:12.321148 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bf62a5_3695_4859_aa0c_cdb4d45e8b7a.slice/crio-7ac08bb8854014531b999e66622cff911484170206f450dd8e1f455a1fed97ca WatchSource:0}: Error finding container 7ac08bb8854014531b999e66622cff911484170206f450dd8e1f455a1fed97ca: Status 404 returned error can't find the container with id 7ac08bb8854014531b999e66622cff911484170206f450dd8e1f455a1fed97ca Sep 07 13:14:12.722670 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:12.722633 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" event={"ID":"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a","Type":"ContainerStarted","Data":"243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0"} Sep 07 13:14:12.723063 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:12.722726 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" event={"ID":"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a","Type":"ContainerStarted","Data":"7ac08bb8854014531b999e66622cff911484170206f450dd8e1f455a1fed97ca"} Sep 07 13:14:12.723063 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:12.722971 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:14:13.674752 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:13.674718 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:14:13.674907 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:13.674802 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:14:13.695115 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:13.695065 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" podStartSLOduration=2.695049787 podStartE2EDuration="2.695049787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:12.748682758 +0000 UTC m=+874.429068528" watchObservedRunningTime="2026-09-07 13:14:13.695049787 +0000 UTC m=+875.375435561" Sep 07 13:14:22.725583 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:22.725549 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:22.726147 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:22.725673 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:24.735787 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:24.735755 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:14:33.734502 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:33.734416 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj"] Sep 07 13:14:33.738660 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:33.738637 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" Sep 07 13:14:33.743252 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:33.743227 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 13:14:33.743384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:33.743320 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jlvcf\"" Sep 07 13:14:33.752681 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:33.752656 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj"] Sep 07 13:14:33.839525 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:33.839499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40aa2a25-b513-4b2b-8253-94d78d933252-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj\" (UID: \"40aa2a25-b513-4b2b-8253-94d78d933252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" Sep 07 13:14:33.940574 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:33.940545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40aa2a25-b513-4b2b-8253-94d78d933252-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj\" (UID: \"40aa2a25-b513-4b2b-8253-94d78d933252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" Sep 07 13:14:33.941129 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:33.941110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40aa2a25-b513-4b2b-8253-94d78d933252-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj\" (UID: \"40aa2a25-b513-4b2b-8253-94d78d933252\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" Sep 07 13:14:34.071683 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:34.071609 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" Sep 07 13:14:34.345410 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:34.345251 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj"] Sep 07 13:14:34.347367 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:14:34.347338 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40aa2a25_b513_4b2b_8253_94d78d933252.slice/crio-8f44ba79ce2a219e6e2861f7dc0b985be150fc2e2f31bf22abfd75a025a051ec WatchSource:0}: Error finding container 8f44ba79ce2a219e6e2861f7dc0b985be150fc2e2f31bf22abfd75a025a051ec: Status 404 returned error can't find the container with id 8f44ba79ce2a219e6e2861f7dc0b985be150fc2e2f31bf22abfd75a025a051ec Sep 07 13:14:34.835112 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:34.835077 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" event={"ID":"40aa2a25-b513-4b2b-8253-94d78d933252","Type":"ContainerStarted","Data":"8f44ba79ce2a219e6e2861f7dc0b985be150fc2e2f31bf22abfd75a025a051ec"} Sep 07 13:14:37.049065 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.049029 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5"] Sep 07 13:14:37.057867 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.057845 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:14:37.060737 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.060713 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-824a79\"" Sep 07 13:14:37.062766 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.062746 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5"] Sep 07 13:14:37.171575 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.171305 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt4wd\" (UniqueName: \"kubernetes.io/projected/ff35f4a6-e889-491a-9366-a65fdcd8b39b-kube-api-access-dt4wd\") pod \"payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5\" (UID: \"ff35f4a6-e889-491a-9366-a65fdcd8b39b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:14:37.171575 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.171406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff35f4a6-e889-491a-9366-a65fdcd8b39b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5\" (UID: \"ff35f4a6-e889-491a-9366-a65fdcd8b39b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:14:37.201589 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.198726 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl"] Sep 07 13:14:37.207269 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.207217 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:14:37.210276 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.210093 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-824a79-dockercfg-ntjn8\"" Sep 07 13:14:37.210870 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.210844 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl"] Sep 07 13:14:37.272730 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.272686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9k8\" (UniqueName: \"kubernetes.io/projected/1715701b-b864-470a-a6fd-122389e69ed8-kube-api-access-lt9k8\") pod \"payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl\" (UID: \"1715701b-b864-470a-a6fd-122389e69ed8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:14:37.272890 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.272768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt4wd\" (UniqueName: \"kubernetes.io/projected/ff35f4a6-e889-491a-9366-a65fdcd8b39b-kube-api-access-dt4wd\") pod \"payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5\" (UID: \"ff35f4a6-e889-491a-9366-a65fdcd8b39b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:14:37.272890 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.272844 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1715701b-b864-470a-a6fd-122389e69ed8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl\" (UID: \"1715701b-b864-470a-a6fd-122389e69ed8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:14:37.272890 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.272864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff35f4a6-e889-491a-9366-a65fdcd8b39b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5\" (UID: \"ff35f4a6-e889-491a-9366-a65fdcd8b39b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:14:37.273647 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.273620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff35f4a6-e889-491a-9366-a65fdcd8b39b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5\" (UID: \"ff35f4a6-e889-491a-9366-a65fdcd8b39b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:14:37.281056 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.281032 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt4wd\" (UniqueName: \"kubernetes.io/projected/ff35f4a6-e889-491a-9366-a65fdcd8b39b-kube-api-access-dt4wd\") pod \"payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5\" (UID: \"ff35f4a6-e889-491a-9366-a65fdcd8b39b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:14:37.372944 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.372828 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:14:37.373363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.373345 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1715701b-b864-470a-a6fd-122389e69ed8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl\" (UID: \"1715701b-b864-470a-a6fd-122389e69ed8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:14:37.373439 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.373421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9k8\" (UniqueName: \"kubernetes.io/projected/1715701b-b864-470a-a6fd-122389e69ed8-kube-api-access-lt9k8\") pod \"payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl\" (UID: \"1715701b-b864-470a-a6fd-122389e69ed8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:14:37.374267 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.374244 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1715701b-b864-470a-a6fd-122389e69ed8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl\" (UID: \"1715701b-b864-470a-a6fd-122389e69ed8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:14:37.382124 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.382098 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9k8\" (UniqueName: \"kubernetes.io/projected/1715701b-b864-470a-a6fd-122389e69ed8-kube-api-access-lt9k8\") pod \"payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl\" (UID: \"1715701b-b864-470a-a6fd-122389e69ed8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:14:37.521387 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.521355 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:14:37.645277 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.645251 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5"] Sep 07 13:14:37.647535 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:14:37.647511 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff35f4a6_e889_491a_9366_a65fdcd8b39b.slice/crio-5d7bc168a4472eb9869349265c89aa8b270be72daac259d185f3f2bc872ec5cb WatchSource:0}: Error finding container 5d7bc168a4472eb9869349265c89aa8b270be72daac259d185f3f2bc872ec5cb: Status 404 returned error can't find the container with id 5d7bc168a4472eb9869349265c89aa8b270be72daac259d185f3f2bc872ec5cb Sep 07 13:14:37.796378 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.796348 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl"] Sep 07 13:14:37.798202 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:14:37.798168 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1715701b_b864_470a_a6fd_122389e69ed8.slice/crio-04873ed12898193d467ef71e3efa9af83f094b70f9497e9459f657feed061556 WatchSource:0}: Error finding container 04873ed12898193d467ef71e3efa9af83f094b70f9497e9459f657feed061556: Status 404 returned error can't find the container with id 04873ed12898193d467ef71e3efa9af83f094b70f9497e9459f657feed061556 Sep 07 13:14:37.852889 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.852865 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" event={"ID":"ff35f4a6-e889-491a-9366-a65fdcd8b39b","Type":"ContainerStarted","Data":"cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e"} Sep 07 13:14:37.852988 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.852919 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" event={"ID":"ff35f4a6-e889-491a-9366-a65fdcd8b39b","Type":"ContainerStarted","Data":"5d7bc168a4472eb9869349265c89aa8b270be72daac259d185f3f2bc872ec5cb"} Sep 07 13:14:37.852988 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.852967 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:14:37.854389 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.854366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" event={"ID":"40aa2a25-b513-4b2b-8253-94d78d933252","Type":"ContainerStarted","Data":"788a13050294f20b3bab1a6ba9ae46ea3aa9890f9812f8caf36c6aaaa651121f"} Sep 07 13:14:37.855516 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.855497 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" event={"ID":"1715701b-b864-470a-a6fd-122389e69ed8","Type":"ContainerStarted","Data":"04873ed12898193d467ef71e3efa9af83f094b70f9497e9459f657feed061556"} Sep 07 13:14:37.871058 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.871014 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" podStartSLOduration=0.870999156 podStartE2EDuration="870.999156ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:37.869395831 +0000 UTC m=+899.549781616" watchObservedRunningTime="2026-09-07 13:14:37.870999156 +0000 UTC m=+899.551384963" Sep 07 13:14:37.885315 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:37.885262 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" podStartSLOduration=1.860724523 podStartE2EDuration="4.885246353s" totalImagesPullingTime="3.02452183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:37.883630077 +0000 UTC m=+899.564015848" watchObservedRunningTime="2026-09-07 13:14:37.885246353 +0000 UTC m=+899.565632150" Sep 07 13:14:38.867406 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:38.867366 2565 generic.go:356] "Generic (PLEG): container finished" podID="40aa2a25-b513-4b2b-8253-94d78d933252" containerID="788a13050294f20b3bab1a6ba9ae46ea3aa9890f9812f8caf36c6aaaa651121f" exitCode=0 Sep 07 13:14:38.867832 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:38.867446 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" event={"ID":"40aa2a25-b513-4b2b-8253-94d78d933252","Type":"ContainerDied","Data":"788a13050294f20b3bab1a6ba9ae46ea3aa9890f9812f8caf36c6aaaa651121f"} Sep 07 13:14:38.869034 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:38.869006 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" event={"ID":"1715701b-b864-470a-a6fd-122389e69ed8","Type":"ContainerStarted","Data":"aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0"} Sep 07 13:14:38.869183 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:38.869077 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:14:38.898868 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:38.898820 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" podStartSLOduration=1.898801861 podStartE2EDuration="1.898801861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:38.898081347 +0000 UTC m=+900.578467121" watchObservedRunningTime="2026-09-07 13:14:38.898801861 +0000 UTC m=+900.579187634" Sep 07 13:14:40.012457 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:40.012435 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" Sep 07 13:14:40.100352 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:40.100324 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40aa2a25-b513-4b2b-8253-94d78d933252-maas-api-service-ca\") pod \"40aa2a25-b513-4b2b-8253-94d78d933252\" (UID: \"40aa2a25-b513-4b2b-8253-94d78d933252\") " Sep 07 13:14:40.100671 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:40.100649 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40aa2a25-b513-4b2b-8253-94d78d933252-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40aa2a25-b513-4b2b-8253-94d78d933252" (UID: "40aa2a25-b513-4b2b-8253-94d78d933252"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:14:40.201259 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:40.201233 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40aa2a25-b513-4b2b-8253-94d78d933252-maas-api-service-ca\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:14:40.881346 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:40.881321 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" Sep 07 13:14:40.881517 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:40.881357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj" event={"ID":"40aa2a25-b513-4b2b-8253-94d78d933252","Type":"ContainerDied","Data":"8f44ba79ce2a219e6e2861f7dc0b985be150fc2e2f31bf22abfd75a025a051ec"} Sep 07 13:14:40.881517 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:40.881401 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f44ba79ce2a219e6e2861f7dc0b985be150fc2e2f31bf22abfd75a025a051ec" Sep 07 13:14:48.871765 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:48.871736 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:14:49.877471 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:49.877440 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:14:50.071761 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.071729 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq"] Sep 07 13:14:50.071954 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.071928 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" podUID="b0ece7cf-c5e3-421c-9a80-43f8ee060e69" containerName="payload-processing" containerID="cri-o://4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125" gracePeriod=30 Sep 07 13:14:50.085662 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.082628 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b"] Sep 07 13:14:50.085662 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.082933 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" podUID="8909b3c6-7932-492d-890c-0dc6ddc3f0ba" containerName="payload-pre-processing" containerID="cri-o://abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57" gracePeriod=30 Sep 07 13:14:50.272276 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.272220 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj"] Sep 07 13:14:50.274210 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.274182 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e07a5db5-qvczj"] Sep 07 13:14:50.360183 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.360161 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:50.363035 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.363015 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:50.496968 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.496886 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dwj7\" (UniqueName: \"kubernetes.io/projected/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-kube-api-access-6dwj7\") pod \"8909b3c6-7932-492d-890c-0dc6ddc3f0ba\" (UID: \"8909b3c6-7932-492d-890c-0dc6ddc3f0ba\") " Sep 07 13:14:50.496968 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.496925 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-plugins-config-volume\") pod \"b0ece7cf-c5e3-421c-9a80-43f8ee060e69\" (UID: \"b0ece7cf-c5e3-421c-9a80-43f8ee060e69\") " Sep 07 13:14:50.497180 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.497043 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-plugins-config-volume\") pod \"8909b3c6-7932-492d-890c-0dc6ddc3f0ba\" (UID: \"8909b3c6-7932-492d-890c-0dc6ddc3f0ba\") " Sep 07 13:14:50.497180 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.497074 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp96p\" (UniqueName: \"kubernetes.io/projected/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-kube-api-access-lp96p\") pod \"b0ece7cf-c5e3-421c-9a80-43f8ee060e69\" (UID: \"b0ece7cf-c5e3-421c-9a80-43f8ee060e69\") " Sep 07 13:14:50.497391 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.497364 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0ece7cf-c5e3-421c-9a80-43f8ee060e69" (UID: "b0ece7cf-c5e3-421c-9a80-43f8ee060e69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:14:50.497440 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.497386 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8909b3c6-7932-492d-890c-0dc6ddc3f0ba" (UID: "8909b3c6-7932-492d-890c-0dc6ddc3f0ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:14:50.499025 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.499001 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-kube-api-access-6dwj7" (OuterVolumeSpecName: "kube-api-access-6dwj7") pod "8909b3c6-7932-492d-890c-0dc6ddc3f0ba" (UID: "8909b3c6-7932-492d-890c-0dc6ddc3f0ba"). InnerVolumeSpecName "kube-api-access-6dwj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:14:50.499133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.499097 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-kube-api-access-lp96p" (OuterVolumeSpecName: "kube-api-access-lp96p") pod "b0ece7cf-c5e3-421c-9a80-43f8ee060e69" (UID: "b0ece7cf-c5e3-421c-9a80-43f8ee060e69"). InnerVolumeSpecName "kube-api-access-lp96p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:14:50.597984 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.597946 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:14:50.597984 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.597982 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp96p\" (UniqueName: \"kubernetes.io/projected/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-kube-api-access-lp96p\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:14:50.597984 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.597992 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwj7\" (UniqueName: \"kubernetes.io/projected/8909b3c6-7932-492d-890c-0dc6ddc3f0ba-kube-api-access-6dwj7\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:14:50.598182 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.598001 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0ece7cf-c5e3-421c-9a80-43f8ee060e69-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:14:50.827061 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.826989 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40aa2a25-b513-4b2b-8253-94d78d933252" path="/var/lib/kubelet/pods/40aa2a25-b513-4b2b-8253-94d78d933252/volumes" Sep 07 13:14:50.933106 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.933071 2565 generic.go:356] "Generic (PLEG): container finished" podID="8909b3c6-7932-492d-890c-0dc6ddc3f0ba" containerID="abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57" exitCode=0 Sep 07 13:14:50.933552 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.933118 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" Sep 07 13:14:50.933552 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.933157 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" event={"ID":"8909b3c6-7932-492d-890c-0dc6ddc3f0ba","Type":"ContainerDied","Data":"abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57"} Sep 07 13:14:50.933552 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.933218 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b" event={"ID":"8909b3c6-7932-492d-890c-0dc6ddc3f0ba","Type":"ContainerDied","Data":"aed921dbb2c8ab8753638a10117c2fc397b6005347b2830f2fca09f6b527c11b"} Sep 07 13:14:50.933552 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.933255 2565 scope.go:117] "RemoveContainer" containerID="abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57" Sep 07 13:14:50.934566 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.934542 2565 generic.go:356] "Generic (PLEG): container finished" podID="b0ece7cf-c5e3-421c-9a80-43f8ee060e69" containerID="4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125" exitCode=0 Sep 07 13:14:50.934704 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.934583 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" event={"ID":"b0ece7cf-c5e3-421c-9a80-43f8ee060e69","Type":"ContainerDied","Data":"4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125"} Sep 07 13:14:50.934704 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.934607 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" Sep 07 13:14:50.934704 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.934635 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq" event={"ID":"b0ece7cf-c5e3-421c-9a80-43f8ee060e69","Type":"ContainerDied","Data":"0f42e8e31e48bb12d5cff047ea06a1b2388debfe1759a4ad8bfb90b12de4047d"} Sep 07 13:14:50.942284 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.942257 2565 scope.go:117] "RemoveContainer" containerID="abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57" Sep 07 13:14:50.942548 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:14:50.942528 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57\": container with ID starting with abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57 not found: ID does not exist" containerID="abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57" Sep 07 13:14:50.942632 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.942556 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57"} err="failed to get container status \"abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57\": rpc error: code = NotFound desc = could not find container \"abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57\": container with ID starting with abca74b85cfa4f2c20e9596088e8ca1dacc941240b7e417212b4560b4ebcdd57 not found: ID does not exist" Sep 07 13:14:50.942632 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.942571 2565 scope.go:117] "RemoveContainer" containerID="4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125" Sep 07 13:14:50.950107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.950091 2565 scope.go:117] "RemoveContainer" containerID="4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125" Sep 07 13:14:50.950402 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:14:50.950378 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125\": container with ID starting with 4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125 not found: ID does not exist" containerID="4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125" Sep 07 13:14:50.950448 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.950408 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125"} err="failed to get container status \"4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125\": rpc error: code = NotFound desc = could not find container \"4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125\": container with ID starting with 4538316802ca1d560728bef2238b33ec4f93c33464f8233d157e13dbd162d125 not found: ID does not exist" Sep 07 13:14:50.955949 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.955928 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b"] Sep 07 13:14:50.961820 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.961800 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e07a5db5-749b6685c8-j5q6b"] Sep 07 13:14:50.975976 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.975957 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq"] Sep 07 13:14:50.985787 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:50.985766 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e07a5db5-68779476d5-z6wfq"] Sep 07 13:14:52.826462 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:52.826433 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8909b3c6-7932-492d-890c-0dc6ddc3f0ba" path="/var/lib/kubelet/pods/8909b3c6-7932-492d-890c-0dc6ddc3f0ba/volumes" Sep 07 13:14:52.826834 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:52.826790 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ece7cf-c5e3-421c-9a80-43f8ee060e69" path="/var/lib/kubelet/pods/b0ece7cf-c5e3-421c-9a80-43f8ee060e69/volumes" Sep 07 13:14:55.119855 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.119823 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc"] Sep 07 13:14:55.120479 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.120459 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40aa2a25-b513-4b2b-8253-94d78d933252" containerName="revoke-keys" Sep 07 13:14:55.120573 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.120482 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="40aa2a25-b513-4b2b-8253-94d78d933252" containerName="revoke-keys" Sep 07 13:14:55.120573 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.120495 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8909b3c6-7932-492d-890c-0dc6ddc3f0ba" containerName="payload-pre-processing" Sep 07 13:14:55.120573 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.120503 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8909b3c6-7932-492d-890c-0dc6ddc3f0ba" containerName="payload-pre-processing" Sep 07 13:14:55.120573 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.120518 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ece7cf-c5e3-421c-9a80-43f8ee060e69" containerName="payload-processing" Sep 07 13:14:55.120573 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.120526 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ece7cf-c5e3-421c-9a80-43f8ee060e69" containerName="payload-processing" Sep 07 13:14:55.120862 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.120638 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8909b3c6-7932-492d-890c-0dc6ddc3f0ba" containerName="payload-pre-processing" Sep 07 13:14:55.120862 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.120650 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ece7cf-c5e3-421c-9a80-43f8ee060e69" containerName="payload-processing" Sep 07 13:14:55.120862 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.120666 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="40aa2a25-b513-4b2b-8253-94d78d933252" containerName="revoke-keys" Sep 07 13:14:55.125144 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.125123 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:14:55.127319 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.127298 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-50f0dc8e-dockercfg-vc244\"" Sep 07 13:14:55.127429 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.127334 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-50f0dc8e\"" Sep 07 13:14:55.132764 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.132739 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc"] Sep 07 13:14:55.193949 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.193914 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft"] Sep 07 13:14:55.197660 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.197613 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:14:55.208423 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.208400 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft"] Sep 07 13:14:55.239905 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.239870 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkg8w\" (UniqueName: \"kubernetes.io/projected/089d2f37-4c00-44aa-b684-9b04783017a3-kube-api-access-qkg8w\") pod \"payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc\" (UID: \"089d2f37-4c00-44aa-b684-9b04783017a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:14:55.240067 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.239926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089d2f37-4c00-44aa-b684-9b04783017a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc\" (UID: \"089d2f37-4c00-44aa-b684-9b04783017a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:14:55.341296 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.341259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089d2f37-4c00-44aa-b684-9b04783017a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc\" (UID: \"089d2f37-4c00-44aa-b684-9b04783017a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:14:55.341460 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.341308 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b84k6\" (UniqueName: \"kubernetes.io/projected/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-kube-api-access-b84k6\") pod \"payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft\" (UID: \"c95ff930-c1db-4e8c-adbb-036e61bbd0c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:14:55.341460 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.341351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft\" (UID: \"c95ff930-c1db-4e8c-adbb-036e61bbd0c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:14:55.341460 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.341386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkg8w\" (UniqueName: \"kubernetes.io/projected/089d2f37-4c00-44aa-b684-9b04783017a3-kube-api-access-qkg8w\") pod \"payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc\" (UID: \"089d2f37-4c00-44aa-b684-9b04783017a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:14:55.341970 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.341950 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089d2f37-4c00-44aa-b684-9b04783017a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc\" (UID: \"089d2f37-4c00-44aa-b684-9b04783017a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:14:55.350374 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.350346 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkg8w\" (UniqueName: \"kubernetes.io/projected/089d2f37-4c00-44aa-b684-9b04783017a3-kube-api-access-qkg8w\") pod \"payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc\" (UID: \"089d2f37-4c00-44aa-b684-9b04783017a3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:14:55.437092 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.437064 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:14:55.442327 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.442309 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84k6\" (UniqueName: \"kubernetes.io/projected/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-kube-api-access-b84k6\") pod \"payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft\" (UID: \"c95ff930-c1db-4e8c-adbb-036e61bbd0c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:14:55.442401 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.442350 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft\" (UID: \"c95ff930-c1db-4e8c-adbb-036e61bbd0c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:14:55.442976 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.442955 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-plugins-config-volume\") pod \"payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft\" (UID: \"c95ff930-c1db-4e8c-adbb-036e61bbd0c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:14:55.451821 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.451795 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b84k6\" (UniqueName: \"kubernetes.io/projected/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-kube-api-access-b84k6\") pod \"payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft\" (UID: \"c95ff930-c1db-4e8c-adbb-036e61bbd0c6\") " pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:14:55.509937 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.509906 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:14:55.711497 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.711473 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc"] Sep 07 13:14:55.713124 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:14:55.713096 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089d2f37_4c00_44aa_b684_9b04783017a3.slice/crio-d46ed438bb0e375654fff496f6d1f4e99f170f5f4ec03e03ba74814abaff7544 WatchSource:0}: Error finding container d46ed438bb0e375654fff496f6d1f4e99f170f5f4ec03e03ba74814abaff7544: Status 404 returned error can't find the container with id d46ed438bb0e375654fff496f6d1f4e99f170f5f4ec03e03ba74814abaff7544 Sep 07 13:14:55.783583 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.783556 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft"] Sep 07 13:14:55.785355 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:14:55.785332 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95ff930_c1db_4e8c_adbb_036e61bbd0c6.slice/crio-22091f3aa9a98bad8731802321ee61394a34f90be52f2a83e707aab30222d227 WatchSource:0}: Error finding container 22091f3aa9a98bad8731802321ee61394a34f90be52f2a83e707aab30222d227: Status 404 returned error can't find the container with id 22091f3aa9a98bad8731802321ee61394a34f90be52f2a83e707aab30222d227 Sep 07 13:14:55.962248 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.962151 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" event={"ID":"c95ff930-c1db-4e8c-adbb-036e61bbd0c6","Type":"ContainerStarted","Data":"95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32"} Sep 07 13:14:55.962412 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.962287 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:14:55.962412 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.962329 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" event={"ID":"c95ff930-c1db-4e8c-adbb-036e61bbd0c6","Type":"ContainerStarted","Data":"22091f3aa9a98bad8731802321ee61394a34f90be52f2a83e707aab30222d227"} Sep 07 13:14:55.963718 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.963695 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" event={"ID":"089d2f37-4c00-44aa-b684-9b04783017a3","Type":"ContainerStarted","Data":"25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542"} Sep 07 13:14:55.963828 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.963735 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" event={"ID":"089d2f37-4c00-44aa-b684-9b04783017a3","Type":"ContainerStarted","Data":"d46ed438bb0e375654fff496f6d1f4e99f170f5f4ec03e03ba74814abaff7544"} Sep 07 13:14:55.963828 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.963754 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:14:55.982021 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.981958 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" podStartSLOduration=0.981939571 podStartE2EDuration="981.939571ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:55.979652288 +0000 UTC m=+917.660038060" watchObservedRunningTime="2026-09-07 13:14:55.981939571 +0000 UTC m=+917.662325343" Sep 07 13:14:55.996543 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:14:55.996494 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" podStartSLOduration=0.996478494 podStartE2EDuration="996.478494ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:14:55.995963823 +0000 UTC m=+917.676349599" watchObservedRunningTime="2026-09-07 13:14:55.996478494 +0000 UTC m=+917.676864268" Sep 07 13:15:00.149320 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.149279 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd"] Sep 07 13:15:00.153723 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.153701 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd" Sep 07 13:15:00.154178 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.154147 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b"] Sep 07 13:15:00.156087 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.156067 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jlvcf\"" Sep 07 13:15:00.157476 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.157456 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b" Sep 07 13:15:00.166338 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.166319 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd" Sep 07 13:15:00.168698 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.168663 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd"] Sep 07 13:15:00.174114 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.174095 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b" Sep 07 13:15:00.184176 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.184078 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b"] Sep 07 13:15:00.444661 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.444638 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd"] Sep 07 13:15:00.446742 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:15:00.446709 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547034c5_f9e1_4561_8c18_dd173f5aa09b.slice/crio-1ff637ec5651a92dfef6d9527ee957b3f4f0ea6986bb83df2832dc4fb88d81a5 WatchSource:0}: Error finding container 1ff637ec5651a92dfef6d9527ee957b3f4f0ea6986bb83df2832dc4fb88d81a5: Status 404 returned error can't find the container with id 1ff637ec5651a92dfef6d9527ee957b3f4f0ea6986bb83df2832dc4fb88d81a5 Sep 07 13:15:00.468078 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.468056 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b"] Sep 07 13:15:00.470033 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:15:00.470010 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35319d4c_ad29_4a89_903a_dff6147ec38f.slice/crio-d9dc3b0237ffc497172b83e7f3aa2bbec903b072f9625a797632135c5feba3dd WatchSource:0}: Error finding container d9dc3b0237ffc497172b83e7f3aa2bbec903b072f9625a797632135c5feba3dd: Status 404 returned error can't find the container with id d9dc3b0237ffc497172b83e7f3aa2bbec903b072f9625a797632135c5feba3dd Sep 07 13:15:00.990327 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.990244 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd" event={"ID":"547034c5-f9e1-4561-8c18-dd173f5aa09b","Type":"ContainerStarted","Data":"d826388d8fc3cee4e783af6943259c754f032676750eddb3b6826e2fdd3589d6"} Sep 07 13:15:00.990327 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.990300 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd" event={"ID":"547034c5-f9e1-4561-8c18-dd173f5aa09b","Type":"ContainerStarted","Data":"1ff637ec5651a92dfef6d9527ee957b3f4f0ea6986bb83df2832dc4fb88d81a5"} Sep 07 13:15:00.991836 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.991811 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b" event={"ID":"35319d4c-ad29-4a89-903a-dff6147ec38f","Type":"ContainerStarted","Data":"f3fdd867264dacc546fd52812a5c23475f942d9e444836653fc945bf19775608"} Sep 07 13:15:00.991927 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:00.991854 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b" event={"ID":"35319d4c-ad29-4a89-903a-dff6147ec38f","Type":"ContainerStarted","Data":"d9dc3b0237ffc497172b83e7f3aa2bbec903b072f9625a797632135c5feba3dd"} Sep 07 13:15:01.009254 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:01.009202 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd" podStartSLOduration=1.009188484 podStartE2EDuration="1.009188484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:15:01.008749642 +0000 UTC m=+922.689135412" watchObservedRunningTime="2026-09-07 13:15:01.009188484 +0000 UTC m=+922.689574280" Sep 07 13:15:01.023696 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:01.023643 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b" podStartSLOduration=1.023627314 podStartE2EDuration="1.023627314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:15:01.023461636 +0000 UTC m=+922.703847407" watchObservedRunningTime="2026-09-07 13:15:01.023627314 +0000 UTC m=+922.704013078" Sep 07 13:15:01.999478 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:01.999445 2565 generic.go:356] "Generic (PLEG): container finished" podID="547034c5-f9e1-4561-8c18-dd173f5aa09b" containerID="d826388d8fc3cee4e783af6943259c754f032676750eddb3b6826e2fdd3589d6" exitCode=0 Sep 07 13:15:01.999923 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:01.999523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd" event={"ID":"547034c5-f9e1-4561-8c18-dd173f5aa09b","Type":"ContainerDied","Data":"d826388d8fc3cee4e783af6943259c754f032676750eddb3b6826e2fdd3589d6"} Sep 07 13:15:02.001413 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:02.001385 2565 generic.go:356] "Generic (PLEG): container finished" podID="35319d4c-ad29-4a89-903a-dff6147ec38f" containerID="f3fdd867264dacc546fd52812a5c23475f942d9e444836653fc945bf19775608" exitCode=0 Sep 07 13:15:02.001546 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:02.001445 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b" event={"ID":"35319d4c-ad29-4a89-903a-dff6147ec38f","Type":"ContainerDied","Data":"f3fdd867264dacc546fd52812a5c23475f942d9e444836653fc945bf19775608"} Sep 07 13:15:03.144733 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:03.144708 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd" Sep 07 13:15:03.181658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:03.181631 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b" Sep 07 13:15:04.014195 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:04.014167 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd" Sep 07 13:15:04.014404 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:04.014164 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd" event={"ID":"547034c5-f9e1-4561-8c18-dd173f5aa09b","Type":"ContainerDied","Data":"1ff637ec5651a92dfef6d9527ee957b3f4f0ea6986bb83df2832dc4fb88d81a5"} Sep 07 13:15:04.014404 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:04.014266 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ff637ec5651a92dfef6d9527ee957b3f4f0ea6986bb83df2832dc4fb88d81a5" Sep 07 13:15:04.015588 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:04.015571 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b" Sep 07 13:15:04.015710 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:04.015613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b" event={"ID":"35319d4c-ad29-4a89-903a-dff6147ec38f","Type":"ContainerDied","Data":"d9dc3b0237ffc497172b83e7f3aa2bbec903b072f9625a797632135c5feba3dd"} Sep 07 13:15:04.015710 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:04.015656 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9dc3b0237ffc497172b83e7f3aa2bbec903b072f9625a797632135c5feba3dd" Sep 07 13:15:06.971178 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:06.971148 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:15:06.971539 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:06.971327 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:15:25.830066 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:25.830032 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft"] Sep 07 13:15:25.832441 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:25.830266 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" podUID="c95ff930-c1db-4e8c-adbb-036e61bbd0c6" containerName="payload-processing" containerID="cri-o://95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32" gracePeriod=30 Sep 07 13:15:25.839022 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:25.838995 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc"] Sep 07 13:15:25.839239 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:25.839214 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" podUID="089d2f37-4c00-44aa-b684-9b04783017a3" containerName="payload-pre-processing" containerID="cri-o://25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542" gracePeriod=30 Sep 07 13:15:26.112995 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.112971 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:15:26.116811 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.116789 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:15:26.130440 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.130407 2565 generic.go:356] "Generic (PLEG): container finished" podID="089d2f37-4c00-44aa-b684-9b04783017a3" containerID="25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542" exitCode=0 Sep 07 13:15:26.130573 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.130455 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" Sep 07 13:15:26.130685 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.130662 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" event={"ID":"089d2f37-4c00-44aa-b684-9b04783017a3","Type":"ContainerDied","Data":"25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542"} Sep 07 13:15:26.130760 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.130737 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc" event={"ID":"089d2f37-4c00-44aa-b684-9b04783017a3","Type":"ContainerDied","Data":"d46ed438bb0e375654fff496f6d1f4e99f170f5f4ec03e03ba74814abaff7544"} Sep 07 13:15:26.130815 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.130791 2565 scope.go:117] "RemoveContainer" containerID="25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542" Sep 07 13:15:26.137820 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.134840 2565 generic.go:356] "Generic (PLEG): container finished" podID="c95ff930-c1db-4e8c-adbb-036e61bbd0c6" containerID="95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32" exitCode=0 Sep 07 13:15:26.137820 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.134880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" event={"ID":"c95ff930-c1db-4e8c-adbb-036e61bbd0c6","Type":"ContainerDied","Data":"95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32"} Sep 07 13:15:26.137820 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.134929 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" event={"ID":"c95ff930-c1db-4e8c-adbb-036e61bbd0c6","Type":"ContainerDied","Data":"22091f3aa9a98bad8731802321ee61394a34f90be52f2a83e707aab30222d227"} Sep 07 13:15:26.137820 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.134991 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft" Sep 07 13:15:26.147440 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.146219 2565 scope.go:117] "RemoveContainer" containerID="25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542" Sep 07 13:15:26.147440 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:15:26.146889 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542\": container with ID starting with 25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542 not found: ID does not exist" containerID="25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542" Sep 07 13:15:26.147440 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.146965 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542"} err="failed to get container status \"25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542\": rpc error: code = NotFound desc = could not find container \"25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542\": container with ID starting with 25df243e458b4af4574b8b15f0d52ec3c6f939f39ec98f2e2b9d90c2147da542 not found: ID does not exist" Sep 07 13:15:26.147440 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.146993 2565 scope.go:117] "RemoveContainer" containerID="95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32" Sep 07 13:15:26.158711 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.156195 2565 scope.go:117] "RemoveContainer" containerID="95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32" Sep 07 13:15:26.158711 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:15:26.156934 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32\": container with ID starting with 95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32 not found: ID does not exist" containerID="95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32" Sep 07 13:15:26.158711 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.156966 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32"} err="failed to get container status \"95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32\": rpc error: code = NotFound desc = could not find container \"95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32\": container with ID starting with 95b84f01850054cc799af57104be04cd1c24be0ec5a29d816fa07490f743eb32 not found: ID does not exist" Sep 07 13:15:26.229326 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.229290 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b84k6\" (UniqueName: \"kubernetes.io/projected/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-kube-api-access-b84k6\") pod \"c95ff930-c1db-4e8c-adbb-036e61bbd0c6\" (UID: \"c95ff930-c1db-4e8c-adbb-036e61bbd0c6\") " Sep 07 13:15:26.229495 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.229362 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkg8w\" (UniqueName: \"kubernetes.io/projected/089d2f37-4c00-44aa-b684-9b04783017a3-kube-api-access-qkg8w\") pod \"089d2f37-4c00-44aa-b684-9b04783017a3\" (UID: \"089d2f37-4c00-44aa-b684-9b04783017a3\") " Sep 07 13:15:26.229495 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.229404 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-plugins-config-volume\") pod \"c95ff930-c1db-4e8c-adbb-036e61bbd0c6\" (UID: \"c95ff930-c1db-4e8c-adbb-036e61bbd0c6\") " Sep 07 13:15:26.229495 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.229452 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089d2f37-4c00-44aa-b684-9b04783017a3-plugins-config-volume\") pod \"089d2f37-4c00-44aa-b684-9b04783017a3\" (UID: \"089d2f37-4c00-44aa-b684-9b04783017a3\") " Sep 07 13:15:26.230265 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.230200 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089d2f37-4c00-44aa-b684-9b04783017a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "089d2f37-4c00-44aa-b684-9b04783017a3" (UID: "089d2f37-4c00-44aa-b684-9b04783017a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:15:26.230265 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.230229 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c95ff930-c1db-4e8c-adbb-036e61bbd0c6" (UID: "c95ff930-c1db-4e8c-adbb-036e61bbd0c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:15:26.237366 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.237338 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089d2f37-4c00-44aa-b684-9b04783017a3-kube-api-access-qkg8w" (OuterVolumeSpecName: "kube-api-access-qkg8w") pod "089d2f37-4c00-44aa-b684-9b04783017a3" (UID: "089d2f37-4c00-44aa-b684-9b04783017a3"). InnerVolumeSpecName "kube-api-access-qkg8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:15:26.237466 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.237377 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-kube-api-access-b84k6" (OuterVolumeSpecName: "kube-api-access-b84k6") pod "c95ff930-c1db-4e8c-adbb-036e61bbd0c6" (UID: "c95ff930-c1db-4e8c-adbb-036e61bbd0c6"). InnerVolumeSpecName "kube-api-access-b84k6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:15:26.330920 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.330882 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b84k6\" (UniqueName: \"kubernetes.io/projected/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-kube-api-access-b84k6\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:15:26.330920 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.330915 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkg8w\" (UniqueName: \"kubernetes.io/projected/089d2f37-4c00-44aa-b684-9b04783017a3-kube-api-access-qkg8w\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:15:26.331097 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.330929 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95ff930-c1db-4e8c-adbb-036e61bbd0c6-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:15:26.331097 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.330942 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089d2f37-4c00-44aa-b684-9b04783017a3-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:15:26.468510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.468476 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc"] Sep 07 13:15:26.480488 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.480456 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-50f0dc8e-77f9459cd-rh7pc"] Sep 07 13:15:26.497324 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.497297 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft"] Sep 07 13:15:26.507377 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.507349 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-50f0dc8e-76964f4ff8-bdpft"] Sep 07 13:15:26.710988 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.710908 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh"] Sep 07 13:15:26.711328 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711316 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c95ff930-c1db-4e8c-adbb-036e61bbd0c6" containerName="payload-processing" Sep 07 13:15:26.711375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711329 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c95ff930-c1db-4e8c-adbb-036e61bbd0c6" containerName="payload-processing" Sep 07 13:15:26.711375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711350 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="089d2f37-4c00-44aa-b684-9b04783017a3" containerName="payload-pre-processing" Sep 07 13:15:26.711375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711355 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="089d2f37-4c00-44aa-b684-9b04783017a3" containerName="payload-pre-processing" Sep 07 13:15:26.711375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711362 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="547034c5-f9e1-4561-8c18-dd173f5aa09b" containerName="cleanup" Sep 07 13:15:26.711375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711367 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="547034c5-f9e1-4561-8c18-dd173f5aa09b" containerName="cleanup" Sep 07 13:15:26.711375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711373 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35319d4c-ad29-4a89-903a-dff6147ec38f" containerName="cleanup" Sep 07 13:15:26.711375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711378 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="35319d4c-ad29-4a89-903a-dff6147ec38f" containerName="cleanup" Sep 07 13:15:26.711588 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711439 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="547034c5-f9e1-4561-8c18-dd173f5aa09b" containerName="cleanup" Sep 07 13:15:26.711588 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711448 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c95ff930-c1db-4e8c-adbb-036e61bbd0c6" containerName="payload-processing" Sep 07 13:15:26.711588 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711455 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="089d2f37-4c00-44aa-b684-9b04783017a3" containerName="payload-pre-processing" Sep 07 13:15:26.711588 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.711461 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="35319d4c-ad29-4a89-903a-dff6147ec38f" containerName="cleanup" Sep 07 13:15:26.715899 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.715852 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.718025 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.718007 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-4aad6a88-kserve-self-signed-certs\"" Sep 07 13:15:26.723367 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.723318 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh"] Sep 07 13:15:26.826932 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.826899 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="089d2f37-4c00-44aa-b684-9b04783017a3" path="/var/lib/kubelet/pods/089d2f37-4c00-44aa-b684-9b04783017a3/volumes" Sep 07 13:15:26.827293 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.827280 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c95ff930-c1db-4e8c-adbb-036e61bbd0c6" path="/var/lib/kubelet/pods/c95ff930-c1db-4e8c-adbb-036e61bbd0c6/volumes" Sep 07 13:15:26.834233 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.834202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-model-cache\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.834702 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.834245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-tmp-dir\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.834702 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.834264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s75q\" (UniqueName: \"kubernetes.io/projected/7285a04d-bf05-4864-b709-8879b726a647-kube-api-access-8s75q\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.834702 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.834444 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-home\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.834702 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.834489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-dshm\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.834702 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.834532 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7285a04d-bf05-4864-b709-8879b726a647-tls-certs\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.935164 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.935127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-home\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.935346 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.935175 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-dshm\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.935414 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.935344 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7285a04d-bf05-4864-b709-8879b726a647-tls-certs\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.935414 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.935397 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-model-cache\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.935527 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.935415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-tmp-dir\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.935527 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.935433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s75q\" (UniqueName: \"kubernetes.io/projected/7285a04d-bf05-4864-b709-8879b726a647-kube-api-access-8s75q\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.935527 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.935487 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-home\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.935808 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.935782 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-model-cache\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.935930 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.935810 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-tmp-dir\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.937731 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.937708 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-dshm\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.938231 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.938211 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7285a04d-bf05-4864-b709-8879b726a647-tls-certs\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:26.951870 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:26.951843 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s75q\" (UniqueName: \"kubernetes.io/projected/7285a04d-bf05-4864-b709-8879b726a647-kube-api-access-8s75q\") pod \"e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:27.026358 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:27.026273 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:27.305345 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:27.305269 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh"] Sep 07 13:15:27.309354 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:15:27.309326 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7285a04d_bf05_4864_b709_8879b726a647.slice/crio-e86f0e7234f36b3d951a9cb01c46a76ce459264c552c3fb6d5a29c856c0f80c8 WatchSource:0}: Error finding container e86f0e7234f36b3d951a9cb01c46a76ce459264c552c3fb6d5a29c856c0f80c8: Status 404 returned error can't find the container with id e86f0e7234f36b3d951a9cb01c46a76ce459264c552c3fb6d5a29c856c0f80c8 Sep 07 13:15:28.150379 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:28.150345 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" event={"ID":"7285a04d-bf05-4864-b709-8879b726a647","Type":"ContainerStarted","Data":"37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86"} Sep 07 13:15:28.150775 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:28.150386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" event={"ID":"7285a04d-bf05-4864-b709-8879b726a647","Type":"ContainerStarted","Data":"e86f0e7234f36b3d951a9cb01c46a76ce459264c552c3fb6d5a29c856c0f80c8"} Sep 07 13:15:28.167902 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:28.167860 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" podStartSLOduration=2.167844918 podStartE2EDuration="2.167844918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:15:28.166703642 +0000 UTC m=+949.847089412" watchObservedRunningTime="2026-09-07 13:15:28.167844918 +0000 UTC m=+949.848230686" Sep 07 13:15:31.131564 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.131524 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr"] Sep 07 13:15:31.135342 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.135323 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:15:31.138920 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.138895 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7d32a524-dockercfg-frc6m\"" Sep 07 13:15:31.139028 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.138941 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7d32a524\"" Sep 07 13:15:31.145921 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.145898 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr"] Sep 07 13:15:31.277585 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.277542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f97f56-7e45-471f-9019-79157cf7a619-plugins-config-volume\") pod \"payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr\" (UID: \"85f97f56-7e45-471f-9019-79157cf7a619\") " pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:15:31.277776 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.277654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74l8c\" (UniqueName: \"kubernetes.io/projected/85f97f56-7e45-471f-9019-79157cf7a619-kube-api-access-74l8c\") pod \"payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr\" (UID: \"85f97f56-7e45-471f-9019-79157cf7a619\") " pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:15:31.378484 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.378445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74l8c\" (UniqueName: \"kubernetes.io/projected/85f97f56-7e45-471f-9019-79157cf7a619-kube-api-access-74l8c\") pod \"payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr\" (UID: \"85f97f56-7e45-471f-9019-79157cf7a619\") " pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:15:31.378715 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.378551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f97f56-7e45-471f-9019-79157cf7a619-plugins-config-volume\") pod \"payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr\" (UID: \"85f97f56-7e45-471f-9019-79157cf7a619\") " pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:15:31.379250 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.379229 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f97f56-7e45-471f-9019-79157cf7a619-plugins-config-volume\") pod \"payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr\" (UID: \"85f97f56-7e45-471f-9019-79157cf7a619\") " pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:15:31.402724 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.402696 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74l8c\" (UniqueName: \"kubernetes.io/projected/85f97f56-7e45-471f-9019-79157cf7a619-kube-api-access-74l8c\") pod \"payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr\" (UID: \"85f97f56-7e45-471f-9019-79157cf7a619\") " pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:15:31.447534 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.447497 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:15:31.724209 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:31.724180 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr"] Sep 07 13:15:31.726242 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:15:31.726214 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f97f56_7e45_471f_9019_79157cf7a619.slice/crio-ac515f113465ea5d7311f278513bb77157031a8da1c843d82de0d411127d34f6 WatchSource:0}: Error finding container ac515f113465ea5d7311f278513bb77157031a8da1c843d82de0d411127d34f6: Status 404 returned error can't find the container with id ac515f113465ea5d7311f278513bb77157031a8da1c843d82de0d411127d34f6 Sep 07 13:15:32.173749 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:32.173715 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" event={"ID":"85f97f56-7e45-471f-9019-79157cf7a619","Type":"ContainerStarted","Data":"a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a"} Sep 07 13:15:32.174123 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:32.173866 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:15:32.174123 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:32.173892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" event={"ID":"85f97f56-7e45-471f-9019-79157cf7a619","Type":"ContainerStarted","Data":"ac515f113465ea5d7311f278513bb77157031a8da1c843d82de0d411127d34f6"} Sep 07 13:15:32.194419 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:32.194365 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" podStartSLOduration=1.1943504520000001 podStartE2EDuration="1.194350452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:15:32.19223444 +0000 UTC m=+953.872620214" watchObservedRunningTime="2026-09-07 13:15:32.194350452 +0000 UTC m=+953.874736266" Sep 07 13:15:37.026845 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:37.026808 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:37.027337 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:37.027087 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:37.039425 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:37.039401 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:37.212282 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:37.212249 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:15:43.181783 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:43.181748 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:15:55.228141 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.228104 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh"] Sep 07 13:15:55.233633 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.233610 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.236645 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.236617 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-adacb1\"/\"openshift-service-ca.crt\"" Sep 07 13:15:55.236784 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.236668 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-adacb1\"/\"autores-model-adacb1-kserve-self-signed-certs\"" Sep 07 13:15:55.236784 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.236630 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-adacb1\"/\"kube-root-ca.crt\"" Sep 07 13:15:55.236784 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.236745 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-adacb1\"/\"default-dockercfg-s6ndw\"" Sep 07 13:15:55.245004 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.244978 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh"] Sep 07 13:15:55.300363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.300327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-tmp-dir\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.300526 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.300376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f607c4-1dbf-4298-8080-d613146e9cbc-tls-certs\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.300526 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.300407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-dshm\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.300526 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.300442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-home\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.300526 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.300466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6658\" (UniqueName: \"kubernetes.io/projected/44f607c4-1dbf-4298-8080-d613146e9cbc-kube-api-access-h6658\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.300695 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.300559 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-model-cache\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.401582 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.401544 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-home\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.401582 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.401588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6658\" (UniqueName: \"kubernetes.io/projected/44f607c4-1dbf-4298-8080-d613146e9cbc-kube-api-access-h6658\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.401831 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.401636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-model-cache\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.401831 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.401664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-tmp-dir\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.401831 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.401799 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f607c4-1dbf-4298-8080-d613146e9cbc-tls-certs\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.401986 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.401873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-dshm\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.401986 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.401945 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-home\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.402056 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.401982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-model-cache\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.402056 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.402027 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-tmp-dir\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.404107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.404085 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-dshm\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.404342 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.404325 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f607c4-1dbf-4298-8080-d613146e9cbc-tls-certs\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.410135 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.410114 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6658\" (UniqueName: \"kubernetes.io/projected/44f607c4-1dbf-4298-8080-d613146e9cbc-kube-api-access-h6658\") pod \"autores-model-adacb1-kserve-fb5dbd4cf-pt6bh\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.542840 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.542749 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:15:55.815716 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:55.815691 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh"] Sep 07 13:15:55.817479 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:15:55.817451 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44f607c4_1dbf_4298_8080_d613146e9cbc.slice/crio-106c647bb1b4736d1f835faeaf8e406b3397f4490d7712e4f386a0099d5c6858 WatchSource:0}: Error finding container 106c647bb1b4736d1f835faeaf8e406b3397f4490d7712e4f386a0099d5c6858: Status 404 returned error can't find the container with id 106c647bb1b4736d1f835faeaf8e406b3397f4490d7712e4f386a0099d5c6858 Sep 07 13:15:56.303137 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:56.303095 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" event={"ID":"44f607c4-1dbf-4298-8080-d613146e9cbc","Type":"ContainerStarted","Data":"bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504"} Sep 07 13:15:56.303585 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:56.303146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" event={"ID":"44f607c4-1dbf-4298-8080-d613146e9cbc","Type":"ContainerStarted","Data":"106c647bb1b4736d1f835faeaf8e406b3397f4490d7712e4f386a0099d5c6858"} Sep 07 13:15:56.322546 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:15:56.322494 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" podStartSLOduration=1.322474378 podStartE2EDuration="1.322474378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:15:56.321710114 +0000 UTC m=+978.002095888" watchObservedRunningTime="2026-09-07 13:15:56.322474378 +0000 UTC m=+978.002860149" Sep 07 13:16:00.231129 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.231097 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j"] Sep 07 13:16:00.234861 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.234842 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.236981 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.236951 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ebfa46\"/\"kube-root-ca.crt\"" Sep 07 13:16:00.237806 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.237784 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ebfa46\"/\"default-dockercfg-n9kg4\"" Sep 07 13:16:00.237971 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.237853 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ebfa46\"/\"test-model-ebfa46-kserve-self-signed-certs\"" Sep 07 13:16:00.238093 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.238076 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ebfa46\"/\"openshift-service-ca.crt\"" Sep 07 13:16:00.242642 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.242619 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j"] Sep 07 13:16:00.346422 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.346386 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-dshm\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.346422 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.346426 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tmp-dir\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.346710 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.346546 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-model-cache\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.346710 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.346578 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tls-certs\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.346710 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.346624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-home\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.346710 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.346671 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn9j5\" (UniqueName: \"kubernetes.io/projected/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-kube-api-access-qn9j5\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.447622 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.447554 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-dshm\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.447622 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.447628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tmp-dir\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.447877 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.447723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-model-cache\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.447877 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.447754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tls-certs\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.447877 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.447776 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-home\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.447877 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.447810 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn9j5\" (UniqueName: \"kubernetes.io/projected/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-kube-api-access-qn9j5\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.448088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.448066 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tmp-dir\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.448155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.448078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-model-cache\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.448155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.448115 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-home\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.449870 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.449849 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-dshm\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.450179 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.450161 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tls-certs\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.456359 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.456337 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn9j5\" (UniqueName: \"kubernetes.io/projected/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-kube-api-access-qn9j5\") pod \"test-model-ebfa46-kserve-7bbd558d45-bpt5j\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.546133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.546056 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:00.832474 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:00.832439 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j"] Sep 07 13:16:00.835205 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:16:00.835169 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod247a5fcd_7e55_4f46_9ef2_9ffb0f71d7eb.slice/crio-f32a8ae918e7fbc4a52547de86cbdcd1c92b087a62aadeae64ea424293caab21 WatchSource:0}: Error finding container f32a8ae918e7fbc4a52547de86cbdcd1c92b087a62aadeae64ea424293caab21: Status 404 returned error can't find the container with id f32a8ae918e7fbc4a52547de86cbdcd1c92b087a62aadeae64ea424293caab21 Sep 07 13:16:01.333669 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.333110 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" event={"ID":"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb","Type":"ContainerStarted","Data":"db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4"} Sep 07 13:16:01.333669 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.333170 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" event={"ID":"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb","Type":"ContainerStarted","Data":"f32a8ae918e7fbc4a52547de86cbdcd1c92b087a62aadeae64ea424293caab21"} Sep 07 13:16:01.346172 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.346136 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr"] Sep 07 13:16:01.346415 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.346365 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" podUID="85f97f56-7e45-471f-9019-79157cf7a619" containerName="payload-processing" containerID="cri-o://a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a" gracePeriod=30 Sep 07 13:16:01.366479 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.365847 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" podStartSLOduration=1.36582629 podStartE2EDuration="1.36582629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:16:01.365051503 +0000 UTC m=+983.045437275" watchObservedRunningTime="2026-09-07 13:16:01.36582629 +0000 UTC m=+983.046212060" Sep 07 13:16:01.625962 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.625930 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:16:01.662117 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.662084 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f97f56-7e45-471f-9019-79157cf7a619-plugins-config-volume\") pod \"85f97f56-7e45-471f-9019-79157cf7a619\" (UID: \"85f97f56-7e45-471f-9019-79157cf7a619\") " Sep 07 13:16:01.662304 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.662189 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74l8c\" (UniqueName: \"kubernetes.io/projected/85f97f56-7e45-471f-9019-79157cf7a619-kube-api-access-74l8c\") pod \"85f97f56-7e45-471f-9019-79157cf7a619\" (UID: \"85f97f56-7e45-471f-9019-79157cf7a619\") " Sep 07 13:16:01.662522 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.662494 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f97f56-7e45-471f-9019-79157cf7a619-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85f97f56-7e45-471f-9019-79157cf7a619" (UID: "85f97f56-7e45-471f-9019-79157cf7a619"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:16:01.664515 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.664479 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f97f56-7e45-471f-9019-79157cf7a619-kube-api-access-74l8c" (OuterVolumeSpecName: "kube-api-access-74l8c") pod "85f97f56-7e45-471f-9019-79157cf7a619" (UID: "85f97f56-7e45-471f-9019-79157cf7a619"). InnerVolumeSpecName "kube-api-access-74l8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:16:01.769120 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.769075 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74l8c\" (UniqueName: \"kubernetes.io/projected/85f97f56-7e45-471f-9019-79157cf7a619-kube-api-access-74l8c\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:01.769120 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:01.769120 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f97f56-7e45-471f-9019-79157cf7a619-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:02.339797 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:02.339757 2565 generic.go:356] "Generic (PLEG): container finished" podID="85f97f56-7e45-471f-9019-79157cf7a619" containerID="a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a" exitCode=0 Sep 07 13:16:02.340239 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:02.339806 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" Sep 07 13:16:02.340239 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:02.339839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" event={"ID":"85f97f56-7e45-471f-9019-79157cf7a619","Type":"ContainerDied","Data":"a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a"} Sep 07 13:16:02.340239 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:02.339892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr" event={"ID":"85f97f56-7e45-471f-9019-79157cf7a619","Type":"ContainerDied","Data":"ac515f113465ea5d7311f278513bb77157031a8da1c843d82de0d411127d34f6"} Sep 07 13:16:02.340239 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:02.339915 2565 scope.go:117] "RemoveContainer" containerID="a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a" Sep 07 13:16:02.349128 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:02.349112 2565 scope.go:117] "RemoveContainer" containerID="a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a" Sep 07 13:16:02.349413 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:16:02.349396 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a\": container with ID starting with a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a not found: ID does not exist" containerID="a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a" Sep 07 13:16:02.349476 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:02.349425 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a"} err="failed to get container status \"a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a\": rpc error: code = NotFound desc = could not find container \"a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a\": container with ID starting with a0942310082401a5dbc582745c7af43a0a581d44e51b49d23c092b0bf693262a not found: ID does not exist" Sep 07 13:16:02.362973 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:02.362950 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr"] Sep 07 13:16:02.367271 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:02.367246 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7d32a524-75cc8cbbbc-d6plr"] Sep 07 13:16:02.829161 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:02.829127 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85f97f56-7e45-471f-9019-79157cf7a619" path="/var/lib/kubelet/pods/85f97f56-7e45-471f-9019-79157cf7a619/volumes" Sep 07 13:16:03.675000 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:03.674966 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh"] Sep 07 13:16:05.543020 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:05.542986 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:16:05.543437 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:05.543184 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:16:05.555709 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:05.555686 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:16:06.373375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:06.373337 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:16:08.220040 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.220001 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k"] Sep 07 13:16:08.220426 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.220413 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f97f56-7e45-471f-9019-79157cf7a619" containerName="payload-processing" Sep 07 13:16:08.220426 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.220425 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f97f56-7e45-471f-9019-79157cf7a619" containerName="payload-processing" Sep 07 13:16:08.220508 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.220498 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f97f56-7e45-471f-9019-79157cf7a619" containerName="payload-processing" Sep 07 13:16:08.223489 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.223474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:08.225660 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.225638 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-bc6fa7f3-dockercfg-pjp47\"" Sep 07 13:16:08.225660 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.225651 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-bc6fa7f3\"" Sep 07 13:16:08.233789 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.233766 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k"] Sep 07 13:16:08.336749 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.336716 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4388835-85dd-49bc-be3c-6fbe3c663434-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k\" (UID: \"c4388835-85dd-49bc-be3c-6fbe3c663434\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:08.336910 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.336770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jns\" (UniqueName: \"kubernetes.io/projected/c4388835-85dd-49bc-be3c-6fbe3c663434-kube-api-access-n4jns\") pod \"payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k\" (UID: \"c4388835-85dd-49bc-be3c-6fbe3c663434\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:08.437863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.437823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4388835-85dd-49bc-be3c-6fbe3c663434-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k\" (UID: \"c4388835-85dd-49bc-be3c-6fbe3c663434\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:08.438059 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.437886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jns\" (UniqueName: \"kubernetes.io/projected/c4388835-85dd-49bc-be3c-6fbe3c663434-kube-api-access-n4jns\") pod \"payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k\" (UID: \"c4388835-85dd-49bc-be3c-6fbe3c663434\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:08.438504 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.438482 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4388835-85dd-49bc-be3c-6fbe3c663434-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k\" (UID: \"c4388835-85dd-49bc-be3c-6fbe3c663434\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:08.454977 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.454955 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jns\" (UniqueName: \"kubernetes.io/projected/c4388835-85dd-49bc-be3c-6fbe3c663434-kube-api-access-n4jns\") pod \"payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k\" (UID: \"c4388835-85dd-49bc-be3c-6fbe3c663434\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:08.534683 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.534584 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:08.809409 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:08.809326 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k"] Sep 07 13:16:08.812852 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:16:08.812826 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4388835_85dd_49bc_be3c_6fbe3c663434.slice/crio-78e253378a84f88b9eacfe4c02e2d421ee5ee0b45f053bc304f835e600705085 WatchSource:0}: Error finding container 78e253378a84f88b9eacfe4c02e2d421ee5ee0b45f053bc304f835e600705085: Status 404 returned error can't find the container with id 78e253378a84f88b9eacfe4c02e2d421ee5ee0b45f053bc304f835e600705085 Sep 07 13:16:09.378178 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:09.378143 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" event={"ID":"c4388835-85dd-49bc-be3c-6fbe3c663434","Type":"ContainerStarted","Data":"d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9"} Sep 07 13:16:09.378556 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:09.378193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" event={"ID":"c4388835-85dd-49bc-be3c-6fbe3c663434","Type":"ContainerStarted","Data":"78e253378a84f88b9eacfe4c02e2d421ee5ee0b45f053bc304f835e600705085"} Sep 07 13:16:09.378556 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:09.378233 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:09.396974 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:09.396927 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" podStartSLOduration=1.396912933 podStartE2EDuration="1.396912933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:16:09.395232674 +0000 UTC m=+991.075618466" watchObservedRunningTime="2026-09-07 13:16:09.396912933 +0000 UTC m=+991.077298703" Sep 07 13:16:10.547167 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:10.547123 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:10.547621 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:10.547184 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:10.560422 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:10.560393 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:11.400915 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:11.400885 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:16:14.640921 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:14.640872 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh"] Sep 07 13:16:18.696785 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:18.696654 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" podUID="7285a04d-bf05-4864-b709-8879b726a647" containerName="main" containerID="cri-o://37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86" gracePeriod=45 Sep 07 13:16:18.946936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:18.946875 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:16:19.047350 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.047321 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7285a04d-bf05-4864-b709-8879b726a647-tls-certs\") pod \"7285a04d-bf05-4864-b709-8879b726a647\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " Sep 07 13:16:19.047550 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.047360 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-home\") pod \"7285a04d-bf05-4864-b709-8879b726a647\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " Sep 07 13:16:19.047550 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.047429 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-model-cache\") pod \"7285a04d-bf05-4864-b709-8879b726a647\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " Sep 07 13:16:19.047550 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.047465 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-tmp-dir\") pod \"7285a04d-bf05-4864-b709-8879b726a647\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " Sep 07 13:16:19.047550 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.047484 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-dshm\") pod \"7285a04d-bf05-4864-b709-8879b726a647\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " Sep 07 13:16:19.047826 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.047613 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s75q\" (UniqueName: \"kubernetes.io/projected/7285a04d-bf05-4864-b709-8879b726a647-kube-api-access-8s75q\") pod \"7285a04d-bf05-4864-b709-8879b726a647\" (UID: \"7285a04d-bf05-4864-b709-8879b726a647\") " Sep 07 13:16:19.047826 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.047663 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-home" (OuterVolumeSpecName: "home") pod "7285a04d-bf05-4864-b709-8879b726a647" (UID: "7285a04d-bf05-4864-b709-8879b726a647"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:16:19.047826 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.047770 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-model-cache" (OuterVolumeSpecName: "model-cache") pod "7285a04d-bf05-4864-b709-8879b726a647" (UID: "7285a04d-bf05-4864-b709-8879b726a647"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:16:19.047826 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.047795 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7285a04d-bf05-4864-b709-8879b726a647" (UID: "7285a04d-bf05-4864-b709-8879b726a647"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:16:19.048037 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.048008 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-model-cache\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:19.048096 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.048053 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-tmp-dir\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:19.048096 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.048067 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-home\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:19.050040 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.049974 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-dshm" (OuterVolumeSpecName: "dshm") pod "7285a04d-bf05-4864-b709-8879b726a647" (UID: "7285a04d-bf05-4864-b709-8879b726a647"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:16:19.050153 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.050043 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7285a04d-bf05-4864-b709-8879b726a647-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7285a04d-bf05-4864-b709-8879b726a647" (UID: "7285a04d-bf05-4864-b709-8879b726a647"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:16:19.050153 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.050046 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7285a04d-bf05-4864-b709-8879b726a647-kube-api-access-8s75q" (OuterVolumeSpecName: "kube-api-access-8s75q") pod "7285a04d-bf05-4864-b709-8879b726a647" (UID: "7285a04d-bf05-4864-b709-8879b726a647"). InnerVolumeSpecName "kube-api-access-8s75q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:16:19.148708 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.148678 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7285a04d-bf05-4864-b709-8879b726a647-dshm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:19.148708 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.148706 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s75q\" (UniqueName: \"kubernetes.io/projected/7285a04d-bf05-4864-b709-8879b726a647-kube-api-access-8s75q\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:19.148878 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.148718 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7285a04d-bf05-4864-b709-8879b726a647-tls-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:19.430042 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.430008 2565 generic.go:356] "Generic (PLEG): container finished" podID="7285a04d-bf05-4864-b709-8879b726a647" containerID="37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86" exitCode=0 Sep 07 13:16:19.430212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.430058 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" Sep 07 13:16:19.430212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.430093 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" event={"ID":"7285a04d-bf05-4864-b709-8879b726a647","Type":"ContainerDied","Data":"37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86"} Sep 07 13:16:19.430212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.430135 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh" event={"ID":"7285a04d-bf05-4864-b709-8879b726a647","Type":"ContainerDied","Data":"e86f0e7234f36b3d951a9cb01c46a76ce459264c552c3fb6d5a29c856c0f80c8"} Sep 07 13:16:19.430212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.430151 2565 scope.go:117] "RemoveContainer" containerID="37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86" Sep 07 13:16:19.439534 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.439516 2565 scope.go:117] "RemoveContainer" containerID="37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86" Sep 07 13:16:19.439812 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:16:19.439793 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86\": container with ID starting with 37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86 not found: ID does not exist" containerID="37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86" Sep 07 13:16:19.439881 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.439818 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86"} err="failed to get container status \"37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86\": rpc error: code = NotFound desc = could not find container \"37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86\": container with ID starting with 37388371070de372cc22a3fc1f6e6c09fd5474ff04ab57ff919557248d54cc86 not found: ID does not exist" Sep 07 13:16:19.455563 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.455536 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh"] Sep 07 13:16:19.458645 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:19.458619 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-4aad6a88-kserve-8667c49b7d-92nxh"] Sep 07 13:16:20.385985 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:20.385954 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:20.828569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:20.828524 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7285a04d-bf05-4864-b709-8879b726a647" path="/var/lib/kubelet/pods/7285a04d-bf05-4864-b709-8879b726a647/volumes" Sep 07 13:16:25.385150 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:25.385113 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd"] Sep 07 13:16:25.387908 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:25.387880 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-adacb1-29813115-62ktd"] Sep 07 13:16:25.415710 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:25.415676 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf"] Sep 07 13:16:25.415966 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:25.415915 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" podUID="21bf62a5-3695-4859-aa0c-cdb4d45e8b7a" containerName="payload-processing" containerID="cri-o://243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0" gracePeriod=30 Sep 07 13:16:25.665757 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:25.665733 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:16:25.813294 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:25.813262 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnckk\" (UniqueName: \"kubernetes.io/projected/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-kube-api-access-wnckk\") pod \"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a\" (UID: \"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a\") " Sep 07 13:16:25.813294 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:25.813300 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-plugins-config-volume\") pod \"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a\" (UID: \"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a\") " Sep 07 13:16:25.813723 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:25.813700 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21bf62a5-3695-4859-aa0c-cdb4d45e8b7a" (UID: "21bf62a5-3695-4859-aa0c-cdb4d45e8b7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:16:25.815307 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:25.815287 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-kube-api-access-wnckk" (OuterVolumeSpecName: "kube-api-access-wnckk") pod "21bf62a5-3695-4859-aa0c-cdb4d45e8b7a" (UID: "21bf62a5-3695-4859-aa0c-cdb4d45e8b7a"). InnerVolumeSpecName "kube-api-access-wnckk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:16:25.914982 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:25.914945 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnckk\" (UniqueName: \"kubernetes.io/projected/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-kube-api-access-wnckk\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:25.914982 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:25.914975 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:26.472768 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:26.472734 2565 generic.go:356] "Generic (PLEG): container finished" podID="21bf62a5-3695-4859-aa0c-cdb4d45e8b7a" containerID="243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0" exitCode=0 Sep 07 13:16:26.473168 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:26.472778 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" Sep 07 13:16:26.473168 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:26.472808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" event={"ID":"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a","Type":"ContainerDied","Data":"243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0"} Sep 07 13:16:26.473168 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:26.472849 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf" event={"ID":"21bf62a5-3695-4859-aa0c-cdb4d45e8b7a","Type":"ContainerDied","Data":"7ac08bb8854014531b999e66622cff911484170206f450dd8e1f455a1fed97ca"} Sep 07 13:16:26.473168 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:26.472870 2565 scope.go:117] "RemoveContainer" containerID="243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0" Sep 07 13:16:26.483556 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:26.483528 2565 scope.go:117] "RemoveContainer" containerID="243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0" Sep 07 13:16:26.491704 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:16:26.488988 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0\": container with ID starting with 243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0 not found: ID does not exist" containerID="243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0" Sep 07 13:16:26.491704 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:26.489041 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0"} err="failed to get container status \"243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0\": rpc error: code = NotFound desc = could not find container \"243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0\": container with ID starting with 243346a99d694593d59d726c9e955adce5e01098f16276a480bef8b06e70a2a0 not found: ID does not exist" Sep 07 13:16:26.495704 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:26.495676 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf"] Sep 07 13:16:26.500011 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:26.499980 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-adacb1-6468f48965-rfbwf"] Sep 07 13:16:26.827136 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:26.827054 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21bf62a5-3695-4859-aa0c-cdb4d45e8b7a" path="/var/lib/kubelet/pods/21bf62a5-3695-4859-aa0c-cdb4d45e8b7a/volumes" Sep 07 13:16:26.827356 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:26.827343 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="547034c5-f9e1-4561-8c18-dd173f5aa09b" path="/var/lib/kubelet/pods/547034c5-f9e1-4561-8c18-dd173f5aa09b/volumes" Sep 07 13:16:29.661291 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:29.661230 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" podUID="44f607c4-1dbf-4298-8080-d613146e9cbc" containerName="main" containerID="cri-o://bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504" gracePeriod=45 Sep 07 13:16:29.913002 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:29.912942 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:16:30.052805 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.052767 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-home\") pod \"44f607c4-1dbf-4298-8080-d613146e9cbc\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " Sep 07 13:16:30.052805 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.052811 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-model-cache\") pod \"44f607c4-1dbf-4298-8080-d613146e9cbc\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " Sep 07 13:16:30.053019 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.052878 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f607c4-1dbf-4298-8080-d613146e9cbc-tls-certs\") pod \"44f607c4-1dbf-4298-8080-d613146e9cbc\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " Sep 07 13:16:30.053019 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.052928 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6658\" (UniqueName: \"kubernetes.io/projected/44f607c4-1dbf-4298-8080-d613146e9cbc-kube-api-access-h6658\") pod \"44f607c4-1dbf-4298-8080-d613146e9cbc\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " Sep 07 13:16:30.053019 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.053006 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-dshm\") pod \"44f607c4-1dbf-4298-8080-d613146e9cbc\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " Sep 07 13:16:30.053145 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.053049 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-tmp-dir\") pod \"44f607c4-1dbf-4298-8080-d613146e9cbc\" (UID: \"44f607c4-1dbf-4298-8080-d613146e9cbc\") " Sep 07 13:16:30.053145 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.053081 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-home" (OuterVolumeSpecName: "home") pod "44f607c4-1dbf-4298-8080-d613146e9cbc" (UID: "44f607c4-1dbf-4298-8080-d613146e9cbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:16:30.053145 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.053096 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "44f607c4-1dbf-4298-8080-d613146e9cbc" (UID: "44f607c4-1dbf-4298-8080-d613146e9cbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:16:30.053323 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.053302 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-home\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:30.053389 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.053327 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-model-cache\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:30.053389 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.053363 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44f607c4-1dbf-4298-8080-d613146e9cbc" (UID: "44f607c4-1dbf-4298-8080-d613146e9cbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:16:30.055160 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.055129 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-dshm" (OuterVolumeSpecName: "dshm") pod "44f607c4-1dbf-4298-8080-d613146e9cbc" (UID: "44f607c4-1dbf-4298-8080-d613146e9cbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:16:30.055249 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.055163 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f607c4-1dbf-4298-8080-d613146e9cbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44f607c4-1dbf-4298-8080-d613146e9cbc" (UID: "44f607c4-1dbf-4298-8080-d613146e9cbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:16:30.055249 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.055224 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f607c4-1dbf-4298-8080-d613146e9cbc-kube-api-access-h6658" (OuterVolumeSpecName: "kube-api-access-h6658") pod "44f607c4-1dbf-4298-8080-d613146e9cbc" (UID: "44f607c4-1dbf-4298-8080-d613146e9cbc"). InnerVolumeSpecName "kube-api-access-h6658". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:16:30.154079 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.154048 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f607c4-1dbf-4298-8080-d613146e9cbc-tls-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:30.154079 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.154076 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6658\" (UniqueName: \"kubernetes.io/projected/44f607c4-1dbf-4298-8080-d613146e9cbc-kube-api-access-h6658\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:30.154079 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.154086 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-dshm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:30.154297 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.154095 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f607c4-1dbf-4298-8080-d613146e9cbc-tmp-dir\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:30.494997 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.494962 2565 generic.go:356] "Generic (PLEG): container finished" podID="44f607c4-1dbf-4298-8080-d613146e9cbc" containerID="bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504" exitCode=0 Sep 07 13:16:30.495173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.495007 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" Sep 07 13:16:30.495173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.495005 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" event={"ID":"44f607c4-1dbf-4298-8080-d613146e9cbc","Type":"ContainerDied","Data":"bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504"} Sep 07 13:16:30.495173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.495106 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh" event={"ID":"44f607c4-1dbf-4298-8080-d613146e9cbc","Type":"ContainerDied","Data":"106c647bb1b4736d1f835faeaf8e406b3397f4490d7712e4f386a0099d5c6858"} Sep 07 13:16:30.495173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.495126 2565 scope.go:117] "RemoveContainer" containerID="bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504" Sep 07 13:16:30.504570 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.504550 2565 scope.go:117] "RemoveContainer" containerID="bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504" Sep 07 13:16:30.504860 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:16:30.504828 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504\": container with ID starting with bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504 not found: ID does not exist" containerID="bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504" Sep 07 13:16:30.504924 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.504867 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504"} err="failed to get container status \"bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504\": rpc error: code = NotFound desc = could not find container \"bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504\": container with ID starting with bf34548be1626c9c4400f37d22ebf6aa52272351a1019a74c2c6f0833758d504 not found: ID does not exist" Sep 07 13:16:30.516844 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.516822 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh"] Sep 07 13:16:30.521606 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.521568 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-adacb1/autores-model-adacb1-kserve-fb5dbd4cf-pt6bh"] Sep 07 13:16:30.827784 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:30.827709 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44f607c4-1dbf-4298-8080-d613146e9cbc" path="/var/lib/kubelet/pods/44f607c4-1dbf-4298-8080-d613146e9cbc/volumes" Sep 07 13:16:38.272409 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.272374 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m"] Sep 07 13:16:38.272906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.272807 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7285a04d-bf05-4864-b709-8879b726a647" containerName="main" Sep 07 13:16:38.272906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.272819 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7285a04d-bf05-4864-b709-8879b726a647" containerName="main" Sep 07 13:16:38.272906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.272826 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21bf62a5-3695-4859-aa0c-cdb4d45e8b7a" containerName="payload-processing" Sep 07 13:16:38.272906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.272831 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="21bf62a5-3695-4859-aa0c-cdb4d45e8b7a" containerName="payload-processing" Sep 07 13:16:38.272906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.272856 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44f607c4-1dbf-4298-8080-d613146e9cbc" containerName="main" Sep 07 13:16:38.272906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.272862 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="44f607c4-1dbf-4298-8080-d613146e9cbc" containerName="main" Sep 07 13:16:38.273102 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.272917 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="44f607c4-1dbf-4298-8080-d613146e9cbc" containerName="main" Sep 07 13:16:38.273102 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.272926 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="21bf62a5-3695-4859-aa0c-cdb4d45e8b7a" containerName="payload-processing" Sep 07 13:16:38.273102 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.272932 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7285a04d-bf05-4864-b709-8879b726a647" containerName="main" Sep 07 13:16:38.275798 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.275782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:16:38.278334 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.278313 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f5e4c4\"" Sep 07 13:16:38.285810 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.285792 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m"] Sep 07 13:16:38.323767 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.323740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fctfw\" (UniqueName: \"kubernetes.io/projected/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-kube-api-access-fctfw\") pod \"payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m\" (UID: \"f31fcd7a-5ef4-4ee9-af74-48959db6d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:16:38.323902 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.323783 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m\" (UID: \"f31fcd7a-5ef4-4ee9-af74-48959db6d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:16:38.424548 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.424509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m\" (UID: \"f31fcd7a-5ef4-4ee9-af74-48959db6d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:16:38.424734 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.424688 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fctfw\" (UniqueName: \"kubernetes.io/projected/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-kube-api-access-fctfw\") pod \"payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m\" (UID: \"f31fcd7a-5ef4-4ee9-af74-48959db6d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:16:38.425429 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.425405 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m\" (UID: \"f31fcd7a-5ef4-4ee9-af74-48959db6d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:16:38.426577 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.426554 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t"] Sep 07 13:16:38.430103 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.430086 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:16:38.432409 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.432390 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f5e4c4-dockercfg-v6n2s\"" Sep 07 13:16:38.439122 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.439102 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fctfw\" (UniqueName: \"kubernetes.io/projected/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-kube-api-access-fctfw\") pod \"payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m\" (UID: \"f31fcd7a-5ef4-4ee9-af74-48959db6d25d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:16:38.445317 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.445295 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t"] Sep 07 13:16:38.525551 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.525460 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwpxc\" (UniqueName: \"kubernetes.io/projected/93bd548a-efb9-4ce8-8251-bbf4d41e0781-kube-api-access-bwpxc\") pod \"payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t\" (UID: \"93bd548a-efb9-4ce8-8251-bbf4d41e0781\") " pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:16:38.525706 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.525589 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93bd548a-efb9-4ce8-8251-bbf4d41e0781-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t\" (UID: \"93bd548a-efb9-4ce8-8251-bbf4d41e0781\") " pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:16:38.587270 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.587221 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:16:38.591798 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.591437 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k"] Sep 07 13:16:38.591798 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.591680 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" podUID="c4388835-85dd-49bc-be3c-6fbe3c663434" containerName="payload-pre-processing" containerID="cri-o://d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9" gracePeriod=30 Sep 07 13:16:38.626259 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.626229 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93bd548a-efb9-4ce8-8251-bbf4d41e0781-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t\" (UID: \"93bd548a-efb9-4ce8-8251-bbf4d41e0781\") " pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:16:38.626381 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.626330 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwpxc\" (UniqueName: \"kubernetes.io/projected/93bd548a-efb9-4ce8-8251-bbf4d41e0781-kube-api-access-bwpxc\") pod \"payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t\" (UID: \"93bd548a-efb9-4ce8-8251-bbf4d41e0781\") " pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:16:38.627162 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.627136 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93bd548a-efb9-4ce8-8251-bbf4d41e0781-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t\" (UID: \"93bd548a-efb9-4ce8-8251-bbf4d41e0781\") " pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:16:38.635489 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.635451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwpxc\" (UniqueName: \"kubernetes.io/projected/93bd548a-efb9-4ce8-8251-bbf4d41e0781-kube-api-access-bwpxc\") pod \"payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t\" (UID: \"93bd548a-efb9-4ce8-8251-bbf4d41e0781\") " pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:16:38.743235 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.743200 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f5e4c4-dockercfg-v6n2s\"" Sep 07 13:16:38.751363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.751320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:16:38.858778 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.858751 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:38.879764 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.879738 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m"] Sep 07 13:16:38.882088 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:16:38.881700 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31fcd7a_5ef4_4ee9_af74_48959db6d25d.slice/crio-8b7518b6a9e1cfebb6e8003b6d810e20c4e7e653b49698c792202acc9c8f16e6 WatchSource:0}: Error finding container 8b7518b6a9e1cfebb6e8003b6d810e20c4e7e653b49698c792202acc9c8f16e6: Status 404 returned error can't find the container with id 8b7518b6a9e1cfebb6e8003b6d810e20c4e7e653b49698c792202acc9c8f16e6 Sep 07 13:16:38.929150 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.929126 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4388835-85dd-49bc-be3c-6fbe3c663434-plugins-config-volume\") pod \"c4388835-85dd-49bc-be3c-6fbe3c663434\" (UID: \"c4388835-85dd-49bc-be3c-6fbe3c663434\") " Sep 07 13:16:38.929264 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.929236 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4jns\" (UniqueName: \"kubernetes.io/projected/c4388835-85dd-49bc-be3c-6fbe3c663434-kube-api-access-n4jns\") pod \"c4388835-85dd-49bc-be3c-6fbe3c663434\" (UID: \"c4388835-85dd-49bc-be3c-6fbe3c663434\") " Sep 07 13:16:38.929567 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.929533 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4388835-85dd-49bc-be3c-6fbe3c663434-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4388835-85dd-49bc-be3c-6fbe3c663434" (UID: "c4388835-85dd-49bc-be3c-6fbe3c663434"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:16:38.931161 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:38.931137 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4388835-85dd-49bc-be3c-6fbe3c663434-kube-api-access-n4jns" (OuterVolumeSpecName: "kube-api-access-n4jns") pod "c4388835-85dd-49bc-be3c-6fbe3c663434" (UID: "c4388835-85dd-49bc-be3c-6fbe3c663434"). InnerVolumeSpecName "kube-api-access-n4jns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:16:39.030589 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.030556 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4388835-85dd-49bc-be3c-6fbe3c663434-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:39.030589 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.030585 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4jns\" (UniqueName: \"kubernetes.io/projected/c4388835-85dd-49bc-be3c-6fbe3c663434-kube-api-access-n4jns\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:16:39.036337 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.036306 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t"] Sep 07 13:16:39.038098 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:16:39.038073 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bd548a_efb9_4ce8_8251_bbf4d41e0781.slice/crio-62911c654415d6df49a450fcbb00e51fc399f4b7cc95b367aa90f36bf16f7687 WatchSource:0}: Error finding container 62911c654415d6df49a450fcbb00e51fc399f4b7cc95b367aa90f36bf16f7687: Status 404 returned error can't find the container with id 62911c654415d6df49a450fcbb00e51fc399f4b7cc95b367aa90f36bf16f7687 Sep 07 13:16:39.554705 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.554655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" event={"ID":"93bd548a-efb9-4ce8-8251-bbf4d41e0781","Type":"ContainerStarted","Data":"3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970"} Sep 07 13:16:39.555123 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.554718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" event={"ID":"93bd548a-efb9-4ce8-8251-bbf4d41e0781","Type":"ContainerStarted","Data":"62911c654415d6df49a450fcbb00e51fc399f4b7cc95b367aa90f36bf16f7687"} Sep 07 13:16:39.555123 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.554749 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:16:39.556113 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.556082 2565 generic.go:356] "Generic (PLEG): container finished" podID="c4388835-85dd-49bc-be3c-6fbe3c663434" containerID="d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9" exitCode=0 Sep 07 13:16:39.556217 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.556120 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" Sep 07 13:16:39.556217 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.556164 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" event={"ID":"c4388835-85dd-49bc-be3c-6fbe3c663434","Type":"ContainerDied","Data":"d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9"} Sep 07 13:16:39.556217 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.556213 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k" event={"ID":"c4388835-85dd-49bc-be3c-6fbe3c663434","Type":"ContainerDied","Data":"78e253378a84f88b9eacfe4c02e2d421ee5ee0b45f053bc304f835e600705085"} Sep 07 13:16:39.556444 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.556238 2565 scope.go:117] "RemoveContainer" containerID="d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9" Sep 07 13:16:39.558058 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.558035 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" event={"ID":"f31fcd7a-5ef4-4ee9-af74-48959db6d25d","Type":"ContainerStarted","Data":"cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc"} Sep 07 13:16:39.558183 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.558082 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" event={"ID":"f31fcd7a-5ef4-4ee9-af74-48959db6d25d","Type":"ContainerStarted","Data":"8b7518b6a9e1cfebb6e8003b6d810e20c4e7e653b49698c792202acc9c8f16e6"} Sep 07 13:16:39.558183 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.558114 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:16:39.566510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.566491 2565 scope.go:117] "RemoveContainer" containerID="d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9" Sep 07 13:16:39.566797 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:16:39.566778 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9\": container with ID starting with d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9 not found: ID does not exist" containerID="d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9" Sep 07 13:16:39.566877 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.566809 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9"} err="failed to get container status \"d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9\": rpc error: code = NotFound desc = could not find container \"d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9\": container with ID starting with d67a6a804fbf19180c0841a67e619a2ad259ef68079ec09a1590012c74ae00b9 not found: ID does not exist" Sep 07 13:16:39.572545 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.572504 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" podStartSLOduration=1.572487499 podStartE2EDuration="1.572487499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:16:39.57145021 +0000 UTC m=+1021.251835980" watchObservedRunningTime="2026-09-07 13:16:39.572487499 +0000 UTC m=+1021.252873271" Sep 07 13:16:39.594243 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.594193 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" podStartSLOduration=1.5941752550000001 podStartE2EDuration="1.594175255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:16:39.591232781 +0000 UTC m=+1021.271618564" watchObservedRunningTime="2026-09-07 13:16:39.594175255 +0000 UTC m=+1021.274561028" Sep 07 13:16:39.606369 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.606341 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k"] Sep 07 13:16:39.611248 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:39.611226 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-bc6fa7f3-67879d495d-5wz5k"] Sep 07 13:16:40.827902 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:40.827870 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4388835-85dd-49bc-be3c-6fbe3c663434" path="/var/lib/kubelet/pods/c4388835-85dd-49bc-be3c-6fbe3c663434/volumes" Sep 07 13:16:50.569068 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:50.569037 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:16:50.569442 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:16:50.569105 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:18:00.364212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:00.364179 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j"] Sep 07 13:18:04.156567 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.156529 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t"] Sep 07 13:18:04.157052 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.156768 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" podUID="93bd548a-efb9-4ce8-8251-bbf4d41e0781" containerName="payload-processing" containerID="cri-o://3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970" gracePeriod=30 Sep 07 13:18:04.165970 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.165941 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m"] Sep 07 13:18:04.166227 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.166174 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" podUID="f31fcd7a-5ef4-4ee9-af74-48959db6d25d" containerName="payload-pre-processing" containerID="cri-o://cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc" gracePeriod=30 Sep 07 13:18:04.418193 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.418133 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:18:04.447419 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.447397 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:18:04.536813 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.536775 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-plugins-config-volume\") pod \"f31fcd7a-5ef4-4ee9-af74-48959db6d25d\" (UID: \"f31fcd7a-5ef4-4ee9-af74-48959db6d25d\") " Sep 07 13:18:04.536994 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.536859 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwpxc\" (UniqueName: \"kubernetes.io/projected/93bd548a-efb9-4ce8-8251-bbf4d41e0781-kube-api-access-bwpxc\") pod \"93bd548a-efb9-4ce8-8251-bbf4d41e0781\" (UID: \"93bd548a-efb9-4ce8-8251-bbf4d41e0781\") " Sep 07 13:18:04.536994 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.536905 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93bd548a-efb9-4ce8-8251-bbf4d41e0781-plugins-config-volume\") pod \"93bd548a-efb9-4ce8-8251-bbf4d41e0781\" (UID: \"93bd548a-efb9-4ce8-8251-bbf4d41e0781\") " Sep 07 13:18:04.536994 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.536936 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fctfw\" (UniqueName: \"kubernetes.io/projected/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-kube-api-access-fctfw\") pod \"f31fcd7a-5ef4-4ee9-af74-48959db6d25d\" (UID: \"f31fcd7a-5ef4-4ee9-af74-48959db6d25d\") " Sep 07 13:18:04.537312 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.537284 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f31fcd7a-5ef4-4ee9-af74-48959db6d25d" (UID: "f31fcd7a-5ef4-4ee9-af74-48959db6d25d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:18:04.537426 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.537358 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bd548a-efb9-4ce8-8251-bbf4d41e0781-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "93bd548a-efb9-4ce8-8251-bbf4d41e0781" (UID: "93bd548a-efb9-4ce8-8251-bbf4d41e0781"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:18:04.539019 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.538997 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bd548a-efb9-4ce8-8251-bbf4d41e0781-kube-api-access-bwpxc" (OuterVolumeSpecName: "kube-api-access-bwpxc") pod "93bd548a-efb9-4ce8-8251-bbf4d41e0781" (UID: "93bd548a-efb9-4ce8-8251-bbf4d41e0781"). InnerVolumeSpecName "kube-api-access-bwpxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:18:04.539327 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.539307 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-kube-api-access-fctfw" (OuterVolumeSpecName: "kube-api-access-fctfw") pod "f31fcd7a-5ef4-4ee9-af74-48959db6d25d" (UID: "f31fcd7a-5ef4-4ee9-af74-48959db6d25d"). InnerVolumeSpecName "kube-api-access-fctfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:18:04.638283 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.638254 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:04.638283 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.638280 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwpxc\" (UniqueName: \"kubernetes.io/projected/93bd548a-efb9-4ce8-8251-bbf4d41e0781-kube-api-access-bwpxc\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:04.638283 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.638289 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93bd548a-efb9-4ce8-8251-bbf4d41e0781-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:04.638506 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.638299 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fctfw\" (UniqueName: \"kubernetes.io/projected/f31fcd7a-5ef4-4ee9-af74-48959db6d25d-kube-api-access-fctfw\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:04.995161 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.995129 2565 generic.go:356] "Generic (PLEG): container finished" podID="f31fcd7a-5ef4-4ee9-af74-48959db6d25d" containerID="cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc" exitCode=0 Sep 07 13:18:04.995311 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.995183 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" Sep 07 13:18:04.995311 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.995202 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" event={"ID":"f31fcd7a-5ef4-4ee9-af74-48959db6d25d","Type":"ContainerDied","Data":"cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc"} Sep 07 13:18:04.995311 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.995273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m" event={"ID":"f31fcd7a-5ef4-4ee9-af74-48959db6d25d","Type":"ContainerDied","Data":"8b7518b6a9e1cfebb6e8003b6d810e20c4e7e653b49698c792202acc9c8f16e6"} Sep 07 13:18:04.995311 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.995298 2565 scope.go:117] "RemoveContainer" containerID="cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc" Sep 07 13:18:04.996526 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.996504 2565 generic.go:356] "Generic (PLEG): container finished" podID="93bd548a-efb9-4ce8-8251-bbf4d41e0781" containerID="3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970" exitCode=0 Sep 07 13:18:04.996621 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.996554 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" Sep 07 13:18:04.996686 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.996632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" event={"ID":"93bd548a-efb9-4ce8-8251-bbf4d41e0781","Type":"ContainerDied","Data":"3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970"} Sep 07 13:18:04.996686 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:04.996681 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t" event={"ID":"93bd548a-efb9-4ce8-8251-bbf4d41e0781","Type":"ContainerDied","Data":"62911c654415d6df49a450fcbb00e51fc399f4b7cc95b367aa90f36bf16f7687"} Sep 07 13:18:05.003741 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:05.003717 2565 scope.go:117] "RemoveContainer" containerID="cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc" Sep 07 13:18:05.003991 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:05.003971 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc\": container with ID starting with cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc not found: ID does not exist" containerID="cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc" Sep 07 13:18:05.004068 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:05.004001 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc"} err="failed to get container status \"cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc\": rpc error: code = NotFound desc = could not find container \"cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc\": container with ID starting with cbd2fe918f9fe568f0aaa1f90632d5aac3d6b4e47e547fd9c439e592ac2381fc not found: ID does not exist" Sep 07 13:18:05.004068 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:05.004024 2565 scope.go:117] "RemoveContainer" containerID="3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970" Sep 07 13:18:05.011811 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:05.011792 2565 scope.go:117] "RemoveContainer" containerID="3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970" Sep 07 13:18:05.012081 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:05.012059 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970\": container with ID starting with 3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970 not found: ID does not exist" containerID="3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970" Sep 07 13:18:05.012168 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:05.012083 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970"} err="failed to get container status \"3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970\": rpc error: code = NotFound desc = could not find container \"3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970\": container with ID starting with 3f2d0fb564b967cc1f10646bf3803b45bf6b4ae984babbf21c4d6c0174951970 not found: ID does not exist" Sep 07 13:18:05.029668 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:05.029644 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m"] Sep 07 13:18:05.037877 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:05.037854 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f5e4c4-dcd4f69cd-fsc8m"] Sep 07 13:18:05.053235 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:05.053213 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t"] Sep 07 13:18:05.070121 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:05.070100 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f5e4c4-7854b94f7c-g9c8t"] Sep 07 13:18:06.827990 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:06.827957 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93bd548a-efb9-4ce8-8251-bbf4d41e0781" path="/var/lib/kubelet/pods/93bd548a-efb9-4ce8-8251-bbf4d41e0781/volumes" Sep 07 13:18:06.828428 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:06.828310 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f31fcd7a-5ef4-4ee9-af74-48959db6d25d" path="/var/lib/kubelet/pods/f31fcd7a-5ef4-4ee9-af74-48959db6d25d/volumes" Sep 07 13:18:12.332326 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.332292 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg"] Sep 07 13:18:12.332812 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.332494 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" podUID="3b54bf30-a16b-48b0-b83d-a6722ca30880" containerName="payload-processing" containerID="cri-o://51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6" gracePeriod=30 Sep 07 13:18:12.345075 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.345048 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d"] Sep 07 13:18:12.345294 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.345257 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" podUID="798fc516-4df3-40a7-b2dc-473d7de484ef" containerName="payload-pre-processing" containerID="cri-o://c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539" gracePeriod=30 Sep 07 13:18:12.622816 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.622788 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:18:12.625661 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.625644 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:18:12.708994 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.708965 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798fc516-4df3-40a7-b2dc-473d7de484ef-plugins-config-volume\") pod \"798fc516-4df3-40a7-b2dc-473d7de484ef\" (UID: \"798fc516-4df3-40a7-b2dc-473d7de484ef\") " Sep 07 13:18:12.709155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.709017 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nghzv\" (UniqueName: \"kubernetes.io/projected/798fc516-4df3-40a7-b2dc-473d7de484ef-kube-api-access-nghzv\") pod \"798fc516-4df3-40a7-b2dc-473d7de484ef\" (UID: \"798fc516-4df3-40a7-b2dc-473d7de484ef\") " Sep 07 13:18:12.709155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.709082 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b54bf30-a16b-48b0-b83d-a6722ca30880-plugins-config-volume\") pod \"3b54bf30-a16b-48b0-b83d-a6722ca30880\" (UID: \"3b54bf30-a16b-48b0-b83d-a6722ca30880\") " Sep 07 13:18:12.709155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.709131 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lvth\" (UniqueName: \"kubernetes.io/projected/3b54bf30-a16b-48b0-b83d-a6722ca30880-kube-api-access-6lvth\") pod \"3b54bf30-a16b-48b0-b83d-a6722ca30880\" (UID: \"3b54bf30-a16b-48b0-b83d-a6722ca30880\") " Sep 07 13:18:12.709380 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.709353 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798fc516-4df3-40a7-b2dc-473d7de484ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "798fc516-4df3-40a7-b2dc-473d7de484ef" (UID: "798fc516-4df3-40a7-b2dc-473d7de484ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:18:12.709497 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.709476 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b54bf30-a16b-48b0-b83d-a6722ca30880-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b54bf30-a16b-48b0-b83d-a6722ca30880" (UID: "3b54bf30-a16b-48b0-b83d-a6722ca30880"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:18:12.711156 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.711137 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798fc516-4df3-40a7-b2dc-473d7de484ef-kube-api-access-nghzv" (OuterVolumeSpecName: "kube-api-access-nghzv") pod "798fc516-4df3-40a7-b2dc-473d7de484ef" (UID: "798fc516-4df3-40a7-b2dc-473d7de484ef"). InnerVolumeSpecName "kube-api-access-nghzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:18:12.711355 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.711334 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b54bf30-a16b-48b0-b83d-a6722ca30880-kube-api-access-6lvth" (OuterVolumeSpecName: "kube-api-access-6lvth") pod "3b54bf30-a16b-48b0-b83d-a6722ca30880" (UID: "3b54bf30-a16b-48b0-b83d-a6722ca30880"). InnerVolumeSpecName "kube-api-access-6lvth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:18:12.810238 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.810208 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798fc516-4df3-40a7-b2dc-473d7de484ef-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:12.810238 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.810232 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nghzv\" (UniqueName: \"kubernetes.io/projected/798fc516-4df3-40a7-b2dc-473d7de484ef-kube-api-access-nghzv\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:12.810389 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.810246 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b54bf30-a16b-48b0-b83d-a6722ca30880-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:12.810389 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:12.810260 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lvth\" (UniqueName: \"kubernetes.io/projected/3b54bf30-a16b-48b0-b83d-a6722ca30880-kube-api-access-6lvth\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:13.038836 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.038801 2565 generic.go:356] "Generic (PLEG): container finished" podID="3b54bf30-a16b-48b0-b83d-a6722ca30880" containerID="51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6" exitCode=0 Sep 07 13:18:13.039016 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.038844 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" Sep 07 13:18:13.039016 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.038887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" event={"ID":"3b54bf30-a16b-48b0-b83d-a6722ca30880","Type":"ContainerDied","Data":"51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6"} Sep 07 13:18:13.039016 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.038941 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg" event={"ID":"3b54bf30-a16b-48b0-b83d-a6722ca30880","Type":"ContainerDied","Data":"d149bfb2db7887f5c813bd74e3f173423204bdfad212907434480071c6ce7022"} Sep 07 13:18:13.039016 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.038969 2565 scope.go:117] "RemoveContainer" containerID="51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6" Sep 07 13:18:13.040254 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.040226 2565 generic.go:356] "Generic (PLEG): container finished" podID="798fc516-4df3-40a7-b2dc-473d7de484ef" containerID="c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539" exitCode=0 Sep 07 13:18:13.040378 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.040285 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" Sep 07 13:18:13.040378 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.040292 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" event={"ID":"798fc516-4df3-40a7-b2dc-473d7de484ef","Type":"ContainerDied","Data":"c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539"} Sep 07 13:18:13.040378 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.040330 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d" event={"ID":"798fc516-4df3-40a7-b2dc-473d7de484ef","Type":"ContainerDied","Data":"2af95d96a61acce376abf35cfe88c98d75de04ba31a4ffb718b103a9d78abcda"} Sep 07 13:18:13.047537 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.047519 2565 scope.go:117] "RemoveContainer" containerID="51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6" Sep 07 13:18:13.047811 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:13.047791 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6\": container with ID starting with 51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6 not found: ID does not exist" containerID="51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6" Sep 07 13:18:13.047857 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.047818 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6"} err="failed to get container status \"51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6\": rpc error: code = NotFound desc = could not find container \"51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6\": container with ID starting with 51b601f9340127efd19cb8a092b859a41125ce2f45c72c29e99e92898f1771c6 not found: ID does not exist" Sep 07 13:18:13.047857 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.047836 2565 scope.go:117] "RemoveContainer" containerID="c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539" Sep 07 13:18:13.055408 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.055391 2565 scope.go:117] "RemoveContainer" containerID="c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539" Sep 07 13:18:13.055696 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:13.055678 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539\": container with ID starting with c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539 not found: ID does not exist" containerID="c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539" Sep 07 13:18:13.055744 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.055702 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539"} err="failed to get container status \"c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539\": rpc error: code = NotFound desc = could not find container \"c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539\": container with ID starting with c77df0ec398a838f6494b980fe8f1147372a2e9b304ba3b9e75adeb9b04b5539 not found: ID does not exist" Sep 07 13:18:13.134569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.134543 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d"] Sep 07 13:18:13.144204 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.144184 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ebfa46-b66959dc4-h477d"] Sep 07 13:18:13.172023 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.171997 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg"] Sep 07 13:18:13.199284 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:13.199261 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ebfa46-556f88c589-cz9hg"] Sep 07 13:18:14.826286 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:14.826251 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b54bf30-a16b-48b0-b83d-a6722ca30880" path="/var/lib/kubelet/pods/3b54bf30-a16b-48b0-b83d-a6722ca30880/volumes" Sep 07 13:18:14.826657 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:14.826621 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="798fc516-4df3-40a7-b2dc-473d7de484ef" path="/var/lib/kubelet/pods/798fc516-4df3-40a7-b2dc-473d7de484ef/volumes" Sep 07 13:18:15.384015 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.383952 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" podUID="247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" containerName="main" containerID="cri-o://db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4" gracePeriod=45 Sep 07 13:18:15.626856 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.626833 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:18:15.735505 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.735473 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn9j5\" (UniqueName: \"kubernetes.io/projected/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-kube-api-access-qn9j5\") pod \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " Sep 07 13:18:15.735696 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.735528 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-home\") pod \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " Sep 07 13:18:15.735696 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.735617 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tmp-dir\") pod \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " Sep 07 13:18:15.735696 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.735655 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tls-certs\") pod \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " Sep 07 13:18:15.735881 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.735698 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-dshm\") pod \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " Sep 07 13:18:15.735881 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.735756 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-model-cache\") pod \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\" (UID: \"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb\") " Sep 07 13:18:15.735881 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.735809 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" (UID: "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:18:15.735881 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.735807 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-home" (OuterVolumeSpecName: "home") pod "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" (UID: "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:18:15.736087 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.736033 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-home\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:15.736087 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.736058 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tmp-dir\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:15.736160 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.736095 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" (UID: "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:18:15.737824 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.737797 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-dshm" (OuterVolumeSpecName: "dshm") pod "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" (UID: "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:18:15.737936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.737860 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-kube-api-access-qn9j5" (OuterVolumeSpecName: "kube-api-access-qn9j5") pod "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" (UID: "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb"). InnerVolumeSpecName "kube-api-access-qn9j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:18:15.737936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.737892 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" (UID: "247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:18:15.837223 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.837195 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-tls-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:15.837223 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.837217 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-dshm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:15.837223 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.837226 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-model-cache\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:15.837590 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:15.837234 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn9j5\" (UniqueName: \"kubernetes.io/projected/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb-kube-api-access-qn9j5\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:16.060765 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:16.060681 2565 generic.go:356] "Generic (PLEG): container finished" podID="247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" containerID="db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4" exitCode=0 Sep 07 13:18:16.060765 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:16.060727 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" Sep 07 13:18:16.060765 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:16.060737 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" event={"ID":"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb","Type":"ContainerDied","Data":"db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4"} Sep 07 13:18:16.060992 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:16.060781 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j" event={"ID":"247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb","Type":"ContainerDied","Data":"f32a8ae918e7fbc4a52547de86cbdcd1c92b087a62aadeae64ea424293caab21"} Sep 07 13:18:16.060992 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:16.060802 2565 scope.go:117] "RemoveContainer" containerID="db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4" Sep 07 13:18:16.069916 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:16.069896 2565 scope.go:117] "RemoveContainer" containerID="db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4" Sep 07 13:18:16.070167 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:16.070145 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4\": container with ID starting with db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4 not found: ID does not exist" containerID="db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4" Sep 07 13:18:16.070210 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:16.070175 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4"} err="failed to get container status \"db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4\": rpc error: code = NotFound desc = could not find container \"db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4\": container with ID starting with db39c857176dc32539709b331c69a6dae7c2475f1bc12f88ecb02e8b5b214db4 not found: ID does not exist" Sep 07 13:18:16.096025 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:16.096000 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j"] Sep 07 13:18:16.098144 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:16.098121 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ebfa46/test-model-ebfa46-kserve-7bbd558d45-bpt5j"] Sep 07 13:18:16.827673 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:16.827639 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" path="/var/lib/kubelet/pods/247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb/volumes" Sep 07 13:18:29.724755 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.724721 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg"] Sep 07 13:18:29.725155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725091 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93bd548a-efb9-4ce8-8251-bbf4d41e0781" containerName="payload-processing" Sep 07 13:18:29.725155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725102 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="93bd548a-efb9-4ce8-8251-bbf4d41e0781" containerName="payload-processing" Sep 07 13:18:29.725155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725115 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="798fc516-4df3-40a7-b2dc-473d7de484ef" containerName="payload-pre-processing" Sep 07 13:18:29.725155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725120 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="798fc516-4df3-40a7-b2dc-473d7de484ef" containerName="payload-pre-processing" Sep 07 13:18:29.725155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725130 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f31fcd7a-5ef4-4ee9-af74-48959db6d25d" containerName="payload-pre-processing" Sep 07 13:18:29.725155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725135 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f31fcd7a-5ef4-4ee9-af74-48959db6d25d" containerName="payload-pre-processing" Sep 07 13:18:29.725155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725145 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b54bf30-a16b-48b0-b83d-a6722ca30880" containerName="payload-processing" Sep 07 13:18:29.725155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725150 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b54bf30-a16b-48b0-b83d-a6722ca30880" containerName="payload-processing" Sep 07 13:18:29.725155 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725158 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" containerName="main" Sep 07 13:18:29.725658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725163 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" containerName="main" Sep 07 13:18:29.725658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725174 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4388835-85dd-49bc-be3c-6fbe3c663434" containerName="payload-pre-processing" Sep 07 13:18:29.725658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725179 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4388835-85dd-49bc-be3c-6fbe3c663434" containerName="payload-pre-processing" Sep 07 13:18:29.725658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725236 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b54bf30-a16b-48b0-b83d-a6722ca30880" containerName="payload-processing" Sep 07 13:18:29.725658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725244 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="798fc516-4df3-40a7-b2dc-473d7de484ef" containerName="payload-pre-processing" Sep 07 13:18:29.725658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725250 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4388835-85dd-49bc-be3c-6fbe3c663434" containerName="payload-pre-processing" Sep 07 13:18:29.725658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725256 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="93bd548a-efb9-4ce8-8251-bbf4d41e0781" containerName="payload-processing" Sep 07 13:18:29.725658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725262 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="247a5fcd-7e55-4f46-9ef2-9ffb0f71d7eb" containerName="main" Sep 07 13:18:29.725658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.725268 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f31fcd7a-5ef4-4ee9-af74-48959db6d25d" containerName="payload-pre-processing" Sep 07 13:18:29.729912 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.729887 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:29.735418 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:29.735382 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg\" is forbidden: User \"system:node:ip-10-0-142-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-216.ec2.internal' and this object" podUID="c3e775c3-2a74-4b99-8674-2a9a08548103" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:29.735564 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:29.735432 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-mt-3dd3e936-dockercfg-kgfb6\" is forbidden: User \"system:node:ip-10-0-142-216.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-216.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3dd3e936-dockercfg-kgfb6\"" type="*v1.Secret" Sep 07 13:18:29.735766 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:29.735741 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-mt-3dd3e936\" is forbidden: User \"system:node:ip-10-0-142-216.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-216.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3dd3e936\"" type="*v1.ConfigMap" Sep 07 13:18:29.778751 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.778723 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg"] Sep 07 13:18:29.830752 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.830721 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz"] Sep 07 13:18:29.834432 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.834415 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:18:29.850465 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.850438 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz"] Sep 07 13:18:29.857861 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.857836 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76qmr\" (UniqueName: \"kubernetes.io/projected/c3e775c3-2a74-4b99-8674-2a9a08548103-kube-api-access-76qmr\") pod \"payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg\" (UID: \"c3e775c3-2a74-4b99-8674-2a9a08548103\") " pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:29.857986 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.857874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e775c3-2a74-4b99-8674-2a9a08548103-plugins-config-volume\") pod \"payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg\" (UID: \"c3e775c3-2a74-4b99-8674-2a9a08548103\") " pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:29.959242 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.959205 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqcj\" (UniqueName: \"kubernetes.io/projected/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-kube-api-access-nrqcj\") pod \"payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz\" (UID: \"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:18:29.959444 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.959281 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76qmr\" (UniqueName: \"kubernetes.io/projected/c3e775c3-2a74-4b99-8674-2a9a08548103-kube-api-access-76qmr\") pod \"payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg\" (UID: \"c3e775c3-2a74-4b99-8674-2a9a08548103\") " pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:29.959444 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.959361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e775c3-2a74-4b99-8674-2a9a08548103-plugins-config-volume\") pod \"payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg\" (UID: \"c3e775c3-2a74-4b99-8674-2a9a08548103\") " pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:29.959580 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:29.959488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz\" (UID: \"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:18:30.025179 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:30.025110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76qmr\" (UniqueName: \"kubernetes.io/projected/c3e775c3-2a74-4b99-8674-2a9a08548103-kube-api-access-76qmr\") pod \"payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg\" (UID: \"c3e775c3-2a74-4b99-8674-2a9a08548103\") " pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:30.060673 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:30.060642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqcj\" (UniqueName: \"kubernetes.io/projected/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-kube-api-access-nrqcj\") pod \"payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz\" (UID: \"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:18:30.060820 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:30.060726 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz\" (UID: \"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:18:30.084247 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:30.084217 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqcj\" (UniqueName: \"kubernetes.io/projected/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-kube-api-access-nrqcj\") pod \"payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz\" (UID: \"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:18:30.960455 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:30.960423 2565 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-mt-3dd3e936: failed to sync configmap cache: timed out waiting for the condition Sep 07 13:18:30.960869 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:30.960521 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c3e775c3-2a74-4b99-8674-2a9a08548103-plugins-config-volume podName:c3e775c3-2a74-4b99-8674-2a9a08548103 nodeName:}" failed. No retries permitted until 2026-09-07 13:18:31.460505219 +0000 UTC m=+1133.140890967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c3e775c3-2a74-4b99-8674-2a9a08548103-plugins-config-volume") pod "payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" (UID: "c3e775c3-2a74-4b99-8674-2a9a08548103") : failed to sync configmap cache: timed out waiting for the condition Sep 07 13:18:31.040220 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:31.040187 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3dd3e936\"" Sep 07 13:18:31.041742 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:31.041722 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz\" (UID: \"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:18:31.295707 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:31.295627 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3dd3e936-dockercfg-kgfb6\"" Sep 07 13:18:31.305429 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:31.305407 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:18:31.472735 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:31.472690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e775c3-2a74-4b99-8674-2a9a08548103-plugins-config-volume\") pod \"payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg\" (UID: \"c3e775c3-2a74-4b99-8674-2a9a08548103\") " pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:31.473345 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:31.473327 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e775c3-2a74-4b99-8674-2a9a08548103-plugins-config-volume\") pod \"payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg\" (UID: \"c3e775c3-2a74-4b99-8674-2a9a08548103\") " pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:31.540558 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:31.540532 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:31.589030 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:31.588999 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz"] Sep 07 13:18:31.829975 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:31.829904 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg"] Sep 07 13:18:31.833429 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:18:31.833399 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e775c3_2a74_4b99_8674_2a9a08548103.slice/crio-5b2c6b2734169a3a7156ac9b7fad9d01449915920b7404d8bcb02e122fc55a0f WatchSource:0}: Error finding container 5b2c6b2734169a3a7156ac9b7fad9d01449915920b7404d8bcb02e122fc55a0f: Status 404 returned error can't find the container with id 5b2c6b2734169a3a7156ac9b7fad9d01449915920b7404d8bcb02e122fc55a0f Sep 07 13:18:32.140130 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:32.140030 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" event={"ID":"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9","Type":"ContainerStarted","Data":"a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5"} Sep 07 13:18:32.140130 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:32.140105 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" event={"ID":"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9","Type":"ContainerStarted","Data":"19a5b1940a8cd2107ffb85271cedd1880e8cfa92eca6d0481fb4d7e31bf07eb1"} Sep 07 13:18:32.140680 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:32.140240 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:18:32.141999 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:32.141976 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" event={"ID":"c3e775c3-2a74-4b99-8674-2a9a08548103","Type":"ContainerStarted","Data":"73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da"} Sep 07 13:18:32.142098 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:32.142019 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" event={"ID":"c3e775c3-2a74-4b99-8674-2a9a08548103","Type":"ContainerStarted","Data":"5b2c6b2734169a3a7156ac9b7fad9d01449915920b7404d8bcb02e122fc55a0f"} Sep 07 13:18:32.142098 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:32.142054 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:32.178162 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:32.178111 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" podStartSLOduration=3.178093391 podStartE2EDuration="3.178093391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:18:32.175325046 +0000 UTC m=+1133.855710818" watchObservedRunningTime="2026-09-07 13:18:32.178093391 +0000 UTC m=+1133.858479164" Sep 07 13:18:32.213241 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:32.213195 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" podStartSLOduration=3.213179751 podStartE2EDuration="3.213179751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:18:32.208905979 +0000 UTC m=+1133.889291748" watchObservedRunningTime="2026-09-07 13:18:32.213179751 +0000 UTC m=+1133.893565527" Sep 07 13:18:35.682761 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:35.682727 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b"] Sep 07 13:18:35.690170 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:35.690146 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-824a79-29813115-4gj7b"] Sep 07 13:18:35.765716 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:35.765684 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl"] Sep 07 13:18:35.765906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:35.765881 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" podUID="1715701b-b864-470a-a6fd-122389e69ed8" containerName="payload-processing" containerID="cri-o://aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0" gracePeriod=30 Sep 07 13:18:35.815326 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:35.815296 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5"] Sep 07 13:18:35.815490 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:35.815472 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" podUID="ff35f4a6-e889-491a-9366-a65fdcd8b39b" containerName="payload-pre-processing" containerID="cri-o://cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e" gracePeriod=30 Sep 07 13:18:36.071573 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.071548 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:18:36.092310 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.092290 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:18:36.164121 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.164092 2565 generic.go:356] "Generic (PLEG): container finished" podID="1715701b-b864-470a-a6fd-122389e69ed8" containerID="aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0" exitCode=0 Sep 07 13:18:36.164276 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.164128 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" Sep 07 13:18:36.164276 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.164168 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" event={"ID":"1715701b-b864-470a-a6fd-122389e69ed8","Type":"ContainerDied","Data":"aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0"} Sep 07 13:18:36.164276 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.164215 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl" event={"ID":"1715701b-b864-470a-a6fd-122389e69ed8","Type":"ContainerDied","Data":"04873ed12898193d467ef71e3efa9af83f094b70f9497e9459f657feed061556"} Sep 07 13:18:36.164276 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.164239 2565 scope.go:117] "RemoveContainer" containerID="aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0" Sep 07 13:18:36.165679 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.165657 2565 generic.go:356] "Generic (PLEG): container finished" podID="ff35f4a6-e889-491a-9366-a65fdcd8b39b" containerID="cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e" exitCode=0 Sep 07 13:18:36.165786 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.165705 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" event={"ID":"ff35f4a6-e889-491a-9366-a65fdcd8b39b","Type":"ContainerDied","Data":"cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e"} Sep 07 13:18:36.165786 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.165714 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" Sep 07 13:18:36.165786 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.165742 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5" event={"ID":"ff35f4a6-e889-491a-9366-a65fdcd8b39b","Type":"ContainerDied","Data":"5d7bc168a4472eb9869349265c89aa8b270be72daac259d185f3f2bc872ec5cb"} Sep 07 13:18:36.174006 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.173988 2565 scope.go:117] "RemoveContainer" containerID="aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0" Sep 07 13:18:36.174621 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:36.174573 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0\": container with ID starting with aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0 not found: ID does not exist" containerID="aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0" Sep 07 13:18:36.174717 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.174631 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0"} err="failed to get container status \"aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0\": rpc error: code = NotFound desc = could not find container \"aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0\": container with ID starting with aabd36458cbffe42d5b6c57b2c86992b6b2aaa39a40fdb303a6fd9941a67dde0 not found: ID does not exist" Sep 07 13:18:36.174717 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.174649 2565 scope.go:117] "RemoveContainer" containerID="cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e" Sep 07 13:18:36.182544 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.182510 2565 scope.go:117] "RemoveContainer" containerID="cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e" Sep 07 13:18:36.182825 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:18:36.182806 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e\": container with ID starting with cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e not found: ID does not exist" containerID="cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e" Sep 07 13:18:36.182901 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.182835 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e"} err="failed to get container status \"cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e\": rpc error: code = NotFound desc = could not find container \"cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e\": container with ID starting with cb58c600cb680cbab1f5af550b29aa9a5f0f14da3c41e9b3c50b384f4b9aab2e not found: ID does not exist" Sep 07 13:18:36.217877 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.217820 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1715701b-b864-470a-a6fd-122389e69ed8-plugins-config-volume\") pod \"1715701b-b864-470a-a6fd-122389e69ed8\" (UID: \"1715701b-b864-470a-a6fd-122389e69ed8\") " Sep 07 13:18:36.217977 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.217899 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt9k8\" (UniqueName: \"kubernetes.io/projected/1715701b-b864-470a-a6fd-122389e69ed8-kube-api-access-lt9k8\") pod \"1715701b-b864-470a-a6fd-122389e69ed8\" (UID: \"1715701b-b864-470a-a6fd-122389e69ed8\") " Sep 07 13:18:36.217977 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.217959 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt4wd\" (UniqueName: \"kubernetes.io/projected/ff35f4a6-e889-491a-9366-a65fdcd8b39b-kube-api-access-dt4wd\") pod \"ff35f4a6-e889-491a-9366-a65fdcd8b39b\" (UID: \"ff35f4a6-e889-491a-9366-a65fdcd8b39b\") " Sep 07 13:18:36.218093 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.217987 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff35f4a6-e889-491a-9366-a65fdcd8b39b-plugins-config-volume\") pod \"ff35f4a6-e889-491a-9366-a65fdcd8b39b\" (UID: \"ff35f4a6-e889-491a-9366-a65fdcd8b39b\") " Sep 07 13:18:36.218173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.218153 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1715701b-b864-470a-a6fd-122389e69ed8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1715701b-b864-470a-a6fd-122389e69ed8" (UID: "1715701b-b864-470a-a6fd-122389e69ed8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:18:36.218411 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.218388 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff35f4a6-e889-491a-9366-a65fdcd8b39b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff35f4a6-e889-491a-9366-a65fdcd8b39b" (UID: "ff35f4a6-e889-491a-9366-a65fdcd8b39b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:18:36.218485 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.218420 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1715701b-b864-470a-a6fd-122389e69ed8-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:36.219807 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.219789 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1715701b-b864-470a-a6fd-122389e69ed8-kube-api-access-lt9k8" (OuterVolumeSpecName: "kube-api-access-lt9k8") pod "1715701b-b864-470a-a6fd-122389e69ed8" (UID: "1715701b-b864-470a-a6fd-122389e69ed8"). InnerVolumeSpecName "kube-api-access-lt9k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:18:36.219914 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.219896 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff35f4a6-e889-491a-9366-a65fdcd8b39b-kube-api-access-dt4wd" (OuterVolumeSpecName: "kube-api-access-dt4wd") pod "ff35f4a6-e889-491a-9366-a65fdcd8b39b" (UID: "ff35f4a6-e889-491a-9366-a65fdcd8b39b"). InnerVolumeSpecName "kube-api-access-dt4wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:18:36.319548 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.319524 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt4wd\" (UniqueName: \"kubernetes.io/projected/ff35f4a6-e889-491a-9366-a65fdcd8b39b-kube-api-access-dt4wd\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:36.319548 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.319545 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff35f4a6-e889-491a-9366-a65fdcd8b39b-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:36.319691 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.319555 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt9k8\" (UniqueName: \"kubernetes.io/projected/1715701b-b864-470a-a6fd-122389e69ed8-kube-api-access-lt9k8\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:18:36.503745 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.503713 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl"] Sep 07 13:18:36.505357 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.505337 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-824a79-6b76cfbc7c-kt9nl"] Sep 07 13:18:36.533566 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.533546 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5"] Sep 07 13:18:36.540788 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.540770 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-824a79-d5d545bfd-gdsc5"] Sep 07 13:18:36.826613 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.826565 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1715701b-b864-470a-a6fd-122389e69ed8" path="/var/lib/kubelet/pods/1715701b-b864-470a-a6fd-122389e69ed8/volumes" Sep 07 13:18:36.826922 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.826881 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35319d4c-ad29-4a89-903a-dff6147ec38f" path="/var/lib/kubelet/pods/35319d4c-ad29-4a89-903a-dff6147ec38f/volumes" Sep 07 13:18:36.827158 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:36.827145 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff35f4a6-e889-491a-9366-a65fdcd8b39b" path="/var/lib/kubelet/pods/ff35f4a6-e889-491a-9366-a65fdcd8b39b/volumes" Sep 07 13:18:43.149219 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:43.149174 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:18:43.149714 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:43.149377 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:18:51.474771 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.474735 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp"] Sep 07 13:18:51.475212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.475088 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1715701b-b864-470a-a6fd-122389e69ed8" containerName="payload-processing" Sep 07 13:18:51.475212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.475099 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1715701b-b864-470a-a6fd-122389e69ed8" containerName="payload-processing" Sep 07 13:18:51.475212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.475125 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff35f4a6-e889-491a-9366-a65fdcd8b39b" containerName="payload-pre-processing" Sep 07 13:18:51.475212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.475131 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff35f4a6-e889-491a-9366-a65fdcd8b39b" containerName="payload-pre-processing" Sep 07 13:18:51.475212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.475195 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff35f4a6-e889-491a-9366-a65fdcd8b39b" containerName="payload-pre-processing" Sep 07 13:18:51.475212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.475208 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1715701b-b864-470a-a6fd-122389e69ed8" containerName="payload-processing" Sep 07 13:18:51.479755 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.479736 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:18:51.487017 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.486996 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-5910bb-dockercfg-wvxr8\"" Sep 07 13:18:51.487542 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.487525 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-5910bb\"" Sep 07 13:18:51.515313 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.515288 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp"] Sep 07 13:18:51.546093 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.546063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02d0975-5818-4498-a1f0-3ddb5d681183-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp\" (UID: \"e02d0975-5818-4498-a1f0-3ddb5d681183\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:18:51.546182 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.546113 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hns8\" (UniqueName: \"kubernetes.io/projected/e02d0975-5818-4498-a1f0-3ddb5d681183-kube-api-access-6hns8\") pod \"payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp\" (UID: \"e02d0975-5818-4498-a1f0-3ddb5d681183\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:18:51.621588 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.621560 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk"] Sep 07 13:18:51.624990 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.624976 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:18:51.646910 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.646884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02d0975-5818-4498-a1f0-3ddb5d681183-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp\" (UID: \"e02d0975-5818-4498-a1f0-3ddb5d681183\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:18:51.647004 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.646925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hns8\" (UniqueName: \"kubernetes.io/projected/e02d0975-5818-4498-a1f0-3ddb5d681183-kube-api-access-6hns8\") pod \"payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp\" (UID: \"e02d0975-5818-4498-a1f0-3ddb5d681183\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:18:51.647004 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.646961 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwdm\" (UniqueName: \"kubernetes.io/projected/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-kube-api-access-lgwdm\") pod \"payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk\" (UID: \"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:18:51.647086 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.647063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk\" (UID: \"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:18:51.647550 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.647532 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02d0975-5818-4498-a1f0-3ddb5d681183-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp\" (UID: \"e02d0975-5818-4498-a1f0-3ddb5d681183\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:18:51.686003 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.685975 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hns8\" (UniqueName: \"kubernetes.io/projected/e02d0975-5818-4498-a1f0-3ddb5d681183-kube-api-access-6hns8\") pod \"payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp\" (UID: \"e02d0975-5818-4498-a1f0-3ddb5d681183\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:18:51.712243 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.712215 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk"] Sep 07 13:18:51.748178 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.748116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwdm\" (UniqueName: \"kubernetes.io/projected/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-kube-api-access-lgwdm\") pod \"payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk\" (UID: \"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:18:51.748303 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.748179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk\" (UID: \"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:18:51.748840 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.748821 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk\" (UID: \"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:18:51.759970 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.759948 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwdm\" (UniqueName: \"kubernetes.io/projected/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-kube-api-access-lgwdm\") pod \"payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk\" (UID: \"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:18:51.789278 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.789248 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:18:51.935754 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:51.935716 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:18:52.065201 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:52.064967 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp"] Sep 07 13:18:52.067291 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:18:52.067265 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02d0975_5818_4498_a1f0_3ddb5d681183.slice/crio-fdc02035805d10b96d739ac392d4158ea074c74a7a3b2821ac34c48d69d3198d WatchSource:0}: Error finding container fdc02035805d10b96d739ac392d4158ea074c74a7a3b2821ac34c48d69d3198d: Status 404 returned error can't find the container with id fdc02035805d10b96d739ac392d4158ea074c74a7a3b2821ac34c48d69d3198d Sep 07 13:18:52.210910 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:52.210886 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk"] Sep 07 13:18:52.213728 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:18:52.213689 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1f7bc1_96a5_4e91_836c_913ffcb2cba1.slice/crio-12e7112b2effffecbd9325d61498f52022c1637b870d5a76c143a137833bf875 WatchSource:0}: Error finding container 12e7112b2effffecbd9325d61498f52022c1637b870d5a76c143a137833bf875: Status 404 returned error can't find the container with id 12e7112b2effffecbd9325d61498f52022c1637b870d5a76c143a137833bf875 Sep 07 13:18:52.245110 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:52.245080 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" event={"ID":"e02d0975-5818-4498-a1f0-3ddb5d681183","Type":"ContainerStarted","Data":"7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743"} Sep 07 13:18:52.245240 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:52.245134 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:18:52.245240 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:52.245155 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" event={"ID":"e02d0975-5818-4498-a1f0-3ddb5d681183","Type":"ContainerStarted","Data":"fdc02035805d10b96d739ac392d4158ea074c74a7a3b2821ac34c48d69d3198d"} Sep 07 13:18:52.246301 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:52.246273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" event={"ID":"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1","Type":"ContainerStarted","Data":"12e7112b2effffecbd9325d61498f52022c1637b870d5a76c143a137833bf875"} Sep 07 13:18:53.252926 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:53.252887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" event={"ID":"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1","Type":"ContainerStarted","Data":"e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0"} Sep 07 13:18:53.253307 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:53.253108 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:18:53.272576 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:53.272529 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" podStartSLOduration=2.272517364 podStartE2EDuration="2.272517364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:18:53.270901216 +0000 UTC m=+1154.951286990" watchObservedRunningTime="2026-09-07 13:18:53.272517364 +0000 UTC m=+1154.952903133" Sep 07 13:18:53.273333 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:18:53.273305 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" podStartSLOduration=2.273297188 podStartE2EDuration="2.273297188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:18:52.262894972 +0000 UTC m=+1153.943280743" watchObservedRunningTime="2026-09-07 13:18:53.273297188 +0000 UTC m=+1154.953683010" Sep 07 13:19:03.255671 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.255586 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:19:03.264159 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.264133 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz"] Sep 07 13:19:03.269043 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.269023 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.271998 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.271970 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-067a48\"/\"openshift-service-ca.crt\"" Sep 07 13:19:03.273078 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.273063 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-067a48\"/\"default-dockercfg-fdbz8\"" Sep 07 13:19:03.273300 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.273283 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-067a48\"/\"kube-root-ca.crt\"" Sep 07 13:19:03.273409 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.273304 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-067a48\"/\"noten-model-067a48-kserve-self-signed-certs\"" Sep 07 13:19:03.291543 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.291523 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz"] Sep 07 13:19:03.359737 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.359702 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-home\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.359918 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.359768 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-dshm\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.359918 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.359792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-model-cache\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.359918 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.359833 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tls-certs\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.359918 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.359873 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp48g\" (UniqueName: \"kubernetes.io/projected/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-kube-api-access-gp48g\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.360082 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.360018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tmp-dir\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.461166 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.461136 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tmp-dir\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.461347 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.461184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-home\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.461347 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.461318 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-dshm\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.461479 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.461368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-model-cache\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.461479 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.461418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tls-certs\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.461479 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.461443 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp48g\" (UniqueName: \"kubernetes.io/projected/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-kube-api-access-gp48g\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.461655 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.461496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-home\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.461655 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.461525 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tmp-dir\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.461767 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.461746 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-model-cache\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.463456 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.463438 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-dshm\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.463794 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.463777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tls-certs\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.474125 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.474105 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp48g\" (UniqueName: \"kubernetes.io/projected/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-kube-api-access-gp48g\") pod \"noten-model-067a48-kserve-5749c6d687-kxnmz\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.578657 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.578547 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:03.855388 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:03.855312 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz"] Sep 07 13:19:03.860158 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:19:03.860129 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2355b4b5_941c_4d2d_ab1a_57ffe05bc789.slice/crio-d031719d27576d82d2c4a0ba313c0aa258af121561438225995fd9dfc418e94b WatchSource:0}: Error finding container d031719d27576d82d2c4a0ba313c0aa258af121561438225995fd9dfc418e94b: Status 404 returned error can't find the container with id d031719d27576d82d2c4a0ba313c0aa258af121561438225995fd9dfc418e94b Sep 07 13:19:04.260024 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:04.259991 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:19:04.308609 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:04.308562 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" event={"ID":"2355b4b5-941c-4d2d-ab1a-57ffe05bc789","Type":"ContainerStarted","Data":"65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb"} Sep 07 13:19:04.308959 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:04.308617 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" event={"ID":"2355b4b5-941c-4d2d-ab1a-57ffe05bc789","Type":"ContainerStarted","Data":"d031719d27576d82d2c4a0ba313c0aa258af121561438225995fd9dfc418e94b"} Sep 07 13:19:04.334150 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:04.334104 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" podStartSLOduration=1.334088445 podStartE2EDuration="1.334088445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:19:04.333544247 +0000 UTC m=+1166.013930020" watchObservedRunningTime="2026-09-07 13:19:04.334088445 +0000 UTC m=+1166.014474214" Sep 07 13:19:13.579230 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:13.579195 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:13.579619 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:13.579251 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:13.592557 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:13.592524 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:14.371556 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:14.371522 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:15.957787 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:15.957754 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg"] Sep 07 13:19:15.958224 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:15.958011 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" podUID="c3e775c3-2a74-4b99-8674-2a9a08548103" containerName="payload-processing" containerID="cri-o://73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da" gracePeriod=30 Sep 07 13:19:15.981774 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:15.981750 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz"] Sep 07 13:19:15.981975 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:15.981952 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" podUID="da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9" containerName="payload-pre-processing" containerID="cri-o://a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5" gracePeriod=30 Sep 07 13:19:16.227110 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.227086 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:19:16.254552 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.254528 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:19:16.370545 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.370512 2565 generic.go:356] "Generic (PLEG): container finished" podID="c3e775c3-2a74-4b99-8674-2a9a08548103" containerID="73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da" exitCode=0 Sep 07 13:19:16.370718 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.370568 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" Sep 07 13:19:16.370718 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.370618 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" event={"ID":"c3e775c3-2a74-4b99-8674-2a9a08548103","Type":"ContainerDied","Data":"73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da"} Sep 07 13:19:16.370718 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.370685 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg" event={"ID":"c3e775c3-2a74-4b99-8674-2a9a08548103","Type":"ContainerDied","Data":"5b2c6b2734169a3a7156ac9b7fad9d01449915920b7404d8bcb02e122fc55a0f"} Sep 07 13:19:16.370718 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.370715 2565 scope.go:117] "RemoveContainer" containerID="73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da" Sep 07 13:19:16.371988 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.371969 2565 generic.go:356] "Generic (PLEG): container finished" podID="da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9" containerID="a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5" exitCode=0 Sep 07 13:19:16.372050 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.372012 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" Sep 07 13:19:16.372050 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.372038 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" event={"ID":"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9","Type":"ContainerDied","Data":"a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5"} Sep 07 13:19:16.372131 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.372070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz" event={"ID":"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9","Type":"ContainerDied","Data":"19a5b1940a8cd2107ffb85271cedd1880e8cfa92eca6d0481fb4d7e31bf07eb1"} Sep 07 13:19:16.379212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.379195 2565 scope.go:117] "RemoveContainer" containerID="73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da" Sep 07 13:19:16.379463 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:19:16.379446 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da\": container with ID starting with 73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da not found: ID does not exist" containerID="73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da" Sep 07 13:19:16.379510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.379472 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da"} err="failed to get container status \"73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da\": rpc error: code = NotFound desc = could not find container \"73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da\": container with ID starting with 73eed011773bd13e178144f478262f662711c890bdf358f232f9f954d9a4a0da not found: ID does not exist" Sep 07 13:19:16.379510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.379489 2565 scope.go:117] "RemoveContainer" containerID="a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5" Sep 07 13:19:16.382655 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.382636 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e775c3-2a74-4b99-8674-2a9a08548103-plugins-config-volume\") pod \"c3e775c3-2a74-4b99-8674-2a9a08548103\" (UID: \"c3e775c3-2a74-4b99-8674-2a9a08548103\") " Sep 07 13:19:16.382750 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.382667 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76qmr\" (UniqueName: \"kubernetes.io/projected/c3e775c3-2a74-4b99-8674-2a9a08548103-kube-api-access-76qmr\") pod \"c3e775c3-2a74-4b99-8674-2a9a08548103\" (UID: \"c3e775c3-2a74-4b99-8674-2a9a08548103\") " Sep 07 13:19:16.382750 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.382696 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-plugins-config-volume\") pod \"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9\" (UID: \"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9\") " Sep 07 13:19:16.382750 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.382729 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrqcj\" (UniqueName: \"kubernetes.io/projected/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-kube-api-access-nrqcj\") pod \"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9\" (UID: \"da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9\") " Sep 07 13:19:16.383037 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.383009 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e775c3-2a74-4b99-8674-2a9a08548103-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3e775c3-2a74-4b99-8674-2a9a08548103" (UID: "c3e775c3-2a74-4b99-8674-2a9a08548103"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:19:16.383126 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.383081 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9" (UID: "da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:19:16.384637 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.384590 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e775c3-2a74-4b99-8674-2a9a08548103-kube-api-access-76qmr" (OuterVolumeSpecName: "kube-api-access-76qmr") pod "c3e775c3-2a74-4b99-8674-2a9a08548103" (UID: "c3e775c3-2a74-4b99-8674-2a9a08548103"). InnerVolumeSpecName "kube-api-access-76qmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:19:16.384793 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.384770 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-kube-api-access-nrqcj" (OuterVolumeSpecName: "kube-api-access-nrqcj") pod "da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9" (UID: "da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9"). InnerVolumeSpecName "kube-api-access-nrqcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:19:16.387222 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.387195 2565 scope.go:117] "RemoveContainer" containerID="a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5" Sep 07 13:19:16.387466 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:19:16.387446 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5\": container with ID starting with a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5 not found: ID does not exist" containerID="a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5" Sep 07 13:19:16.387510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.387473 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5"} err="failed to get container status \"a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5\": rpc error: code = NotFound desc = could not find container \"a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5\": container with ID starting with a29004d4826afc6d6b47efb9b33e0f5642403662c9da1e06a261468bf94426e5 not found: ID does not exist" Sep 07 13:19:16.483508 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.483442 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3e775c3-2a74-4b99-8674-2a9a08548103-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:19:16.483508 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.483465 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76qmr\" (UniqueName: \"kubernetes.io/projected/c3e775c3-2a74-4b99-8674-2a9a08548103-kube-api-access-76qmr\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:19:16.483508 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.483479 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:19:16.483508 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.483491 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqcj\" (UniqueName: \"kubernetes.io/projected/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9-kube-api-access-nrqcj\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:19:16.698797 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.698769 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg"] Sep 07 13:19:16.702798 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.702777 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3dd3e936-5f67bc549-h5hlg"] Sep 07 13:19:16.717366 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.717342 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz"] Sep 07 13:19:16.720217 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.720195 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3dd3e936-7448d8cfd-sl5kz"] Sep 07 13:19:16.826510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.826436 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3e775c3-2a74-4b99-8674-2a9a08548103" path="/var/lib/kubelet/pods/c3e775c3-2a74-4b99-8674-2a9a08548103/volumes" Sep 07 13:19:16.826878 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:16.826863 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9" path="/var/lib/kubelet/pods/da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9/volumes" Sep 07 13:19:26.105378 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.105339 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th"] Sep 07 13:19:26.106818 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.106791 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9" containerName="payload-pre-processing" Sep 07 13:19:26.106974 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.106962 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9" containerName="payload-pre-processing" Sep 07 13:19:26.107088 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.107077 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e775c3-2a74-4b99-8674-2a9a08548103" containerName="payload-processing" Sep 07 13:19:26.107158 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.107149 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e775c3-2a74-4b99-8674-2a9a08548103" containerName="payload-processing" Sep 07 13:19:26.107396 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.107382 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2c6235-b3dc-4c7e-8df2-3d78b9d5c4d9" containerName="payload-pre-processing" Sep 07 13:19:26.107509 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.107499 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e775c3-2a74-4b99-8674-2a9a08548103" containerName="payload-processing" Sep 07 13:19:26.113527 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.113501 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:19:26.121704 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.121679 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c7c371\"" Sep 07 13:19:26.122145 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.122128 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c7c371-dockercfg-v2c9h\"" Sep 07 13:19:26.133375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.133350 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th"] Sep 07 13:19:26.169032 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.169007 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1737bdfa-7386-4c9b-92fe-757724bcf4b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th\" (UID: \"1737bdfa-7386-4c9b-92fe-757724bcf4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:19:26.169144 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.169074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlf4h\" (UniqueName: \"kubernetes.io/projected/1737bdfa-7386-4c9b-92fe-757724bcf4b6-kube-api-access-nlf4h\") pod \"payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th\" (UID: \"1737bdfa-7386-4c9b-92fe-757724bcf4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:19:26.270438 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.270413 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlf4h\" (UniqueName: \"kubernetes.io/projected/1737bdfa-7386-4c9b-92fe-757724bcf4b6-kube-api-access-nlf4h\") pod \"payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th\" (UID: \"1737bdfa-7386-4c9b-92fe-757724bcf4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:19:26.270543 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.270484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1737bdfa-7386-4c9b-92fe-757724bcf4b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th\" (UID: \"1737bdfa-7386-4c9b-92fe-757724bcf4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:19:26.271134 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.271113 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1737bdfa-7386-4c9b-92fe-757724bcf4b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th\" (UID: \"1737bdfa-7386-4c9b-92fe-757724bcf4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:19:26.291301 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.291277 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlf4h\" (UniqueName: \"kubernetes.io/projected/1737bdfa-7386-4c9b-92fe-757724bcf4b6-kube-api-access-nlf4h\") pod \"payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th\" (UID: \"1737bdfa-7386-4c9b-92fe-757724bcf4b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:19:26.396701 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.396641 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl"] Sep 07 13:19:26.400187 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.400169 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:19:26.420640 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.420618 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl"] Sep 07 13:19:26.423449 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.423428 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:19:26.472259 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.472230 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbf37ed-8edf-4104-8080-d5748bb9dacc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl\" (UID: \"acbf37ed-8edf-4104-8080-d5748bb9dacc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:19:26.472427 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.472347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n64rj\" (UniqueName: \"kubernetes.io/projected/acbf37ed-8edf-4104-8080-d5748bb9dacc-kube-api-access-n64rj\") pod \"payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl\" (UID: \"acbf37ed-8edf-4104-8080-d5748bb9dacc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:19:26.573926 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.573892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbf37ed-8edf-4104-8080-d5748bb9dacc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl\" (UID: \"acbf37ed-8edf-4104-8080-d5748bb9dacc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:19:26.573926 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.573932 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n64rj\" (UniqueName: \"kubernetes.io/projected/acbf37ed-8edf-4104-8080-d5748bb9dacc-kube-api-access-n64rj\") pod \"payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl\" (UID: \"acbf37ed-8edf-4104-8080-d5748bb9dacc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:19:26.574569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.574549 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbf37ed-8edf-4104-8080-d5748bb9dacc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl\" (UID: \"acbf37ed-8edf-4104-8080-d5748bb9dacc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:19:26.584981 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.584952 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n64rj\" (UniqueName: \"kubernetes.io/projected/acbf37ed-8edf-4104-8080-d5748bb9dacc-kube-api-access-n64rj\") pod \"payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl\" (UID: \"acbf37ed-8edf-4104-8080-d5748bb9dacc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:19:26.700886 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.700859 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th"] Sep 07 13:19:26.703557 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:19:26.703529 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1737bdfa_7386_4c9b_92fe_757724bcf4b6.slice/crio-6882bce6c6e408471a79029554b2a8184954042fcbc16e4a39c71e6603493e71 WatchSource:0}: Error finding container 6882bce6c6e408471a79029554b2a8184954042fcbc16e4a39c71e6603493e71: Status 404 returned error can't find the container with id 6882bce6c6e408471a79029554b2a8184954042fcbc16e4a39c71e6603493e71 Sep 07 13:19:26.710522 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.710502 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:19:26.997495 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:26.997424 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl"] Sep 07 13:19:27.005954 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:19:27.005930 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbf37ed_8edf_4104_8080_d5748bb9dacc.slice/crio-0877ce1729eaf7fa2c1e390e2efd33f73c6da62f0d9d884ef2f0131b4ad28e60 WatchSource:0}: Error finding container 0877ce1729eaf7fa2c1e390e2efd33f73c6da62f0d9d884ef2f0131b4ad28e60: Status 404 returned error can't find the container with id 0877ce1729eaf7fa2c1e390e2efd33f73c6da62f0d9d884ef2f0131b4ad28e60 Sep 07 13:19:27.431397 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:27.431362 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" event={"ID":"acbf37ed-8edf-4104-8080-d5748bb9dacc","Type":"ContainerStarted","Data":"2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d"} Sep 07 13:19:27.431824 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:27.431420 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" event={"ID":"acbf37ed-8edf-4104-8080-d5748bb9dacc","Type":"ContainerStarted","Data":"0877ce1729eaf7fa2c1e390e2efd33f73c6da62f0d9d884ef2f0131b4ad28e60"} Sep 07 13:19:27.431824 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:27.431451 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:19:27.433227 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:27.433200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" event={"ID":"1737bdfa-7386-4c9b-92fe-757724bcf4b6","Type":"ContainerStarted","Data":"412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d"} Sep 07 13:19:27.433356 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:27.433247 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" event={"ID":"1737bdfa-7386-4c9b-92fe-757724bcf4b6","Type":"ContainerStarted","Data":"6882bce6c6e408471a79029554b2a8184954042fcbc16e4a39c71e6603493e71"} Sep 07 13:19:27.433356 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:27.433286 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:19:27.470197 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:27.470156 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" podStartSLOduration=1.470141757 podStartE2EDuration="1.470141757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:19:27.466796518 +0000 UTC m=+1189.147182289" watchObservedRunningTime="2026-09-07 13:19:27.470141757 +0000 UTC m=+1189.150527527" Sep 07 13:19:27.492245 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:27.492198 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" podStartSLOduration=1.492186878 podStartE2EDuration="1.492186878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:19:27.489692538 +0000 UTC m=+1189.170078334" watchObservedRunningTime="2026-09-07 13:19:27.492186878 +0000 UTC m=+1189.172572649" Sep 07 13:19:36.712997 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:36.712966 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz"] Sep 07 13:19:38.440789 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:38.440761 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:19:38.441151 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:38.440910 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:19:45.616516 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.616481 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6"] Sep 07 13:19:45.621276 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.621260 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:19:45.624562 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.624544 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d5da3f-dockercfg-bh59w\"" Sep 07 13:19:45.624901 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.624879 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d5da3f\"" Sep 07 13:19:45.632772 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.632751 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6"] Sep 07 13:19:45.744574 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.744539 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5042b6d7-912b-4ef1-ace2-5446a290c71d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6\" (UID: \"5042b6d7-912b-4ef1-ace2-5446a290c71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:19:45.744574 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.744574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcgrq\" (UniqueName: \"kubernetes.io/projected/5042b6d7-912b-4ef1-ace2-5446a290c71d-kube-api-access-zcgrq\") pod \"payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6\" (UID: \"5042b6d7-912b-4ef1-ace2-5446a290c71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:19:45.845268 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.845229 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5042b6d7-912b-4ef1-ace2-5446a290c71d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6\" (UID: \"5042b6d7-912b-4ef1-ace2-5446a290c71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:19:45.845268 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.845270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgrq\" (UniqueName: \"kubernetes.io/projected/5042b6d7-912b-4ef1-ace2-5446a290c71d-kube-api-access-zcgrq\") pod \"payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6\" (UID: \"5042b6d7-912b-4ef1-ace2-5446a290c71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:19:45.845926 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.845906 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5042b6d7-912b-4ef1-ace2-5446a290c71d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6\" (UID: \"5042b6d7-912b-4ef1-ace2-5446a290c71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:19:45.855171 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.855151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcgrq\" (UniqueName: \"kubernetes.io/projected/5042b6d7-912b-4ef1-ace2-5446a290c71d-kube-api-access-zcgrq\") pod \"payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6\" (UID: \"5042b6d7-912b-4ef1-ace2-5446a290c71d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:19:45.932004 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:45.931975 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:19:46.207301 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:46.207277 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6"] Sep 07 13:19:46.209256 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:19:46.209229 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5042b6d7_912b_4ef1_ace2_5446a290c71d.slice/crio-93bf02f6d22cda378b15582d5e8e1be515fe7ecb5770a1f41bbd6349d5554c36 WatchSource:0}: Error finding container 93bf02f6d22cda378b15582d5e8e1be515fe7ecb5770a1f41bbd6349d5554c36: Status 404 returned error can't find the container with id 93bf02f6d22cda378b15582d5e8e1be515fe7ecb5770a1f41bbd6349d5554c36 Sep 07 13:19:46.539610 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:46.539508 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" event={"ID":"5042b6d7-912b-4ef1-ace2-5446a290c71d","Type":"ContainerStarted","Data":"ccae08fecf1007b6b5a9cbf1d23c8def84c1a32b13ee9861302aa47a676dda1c"} Sep 07 13:19:46.539610 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:46.539585 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:19:46.539794 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:46.539638 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" event={"ID":"5042b6d7-912b-4ef1-ace2-5446a290c71d","Type":"ContainerStarted","Data":"93bf02f6d22cda378b15582d5e8e1be515fe7ecb5770a1f41bbd6349d5554c36"} Sep 07 13:19:46.565800 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:46.565752 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" podStartSLOduration=1.565738129 podStartE2EDuration="1.565738129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:19:46.564027987 +0000 UTC m=+1208.244413758" watchObservedRunningTime="2026-09-07 13:19:46.565738129 +0000 UTC m=+1208.246123899" Sep 07 13:19:51.735386 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:51.735345 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" podUID="2355b4b5-941c-4d2d-ab1a-57ffe05bc789" containerName="main" containerID="cri-o://65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb" gracePeriod=45 Sep 07 13:19:51.994012 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:51.993958 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:52.097423 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.097395 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-home\") pod \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " Sep 07 13:19:52.097619 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.097461 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-dshm\") pod \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " Sep 07 13:19:52.097619 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.097506 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tls-certs\") pod \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " Sep 07 13:19:52.097619 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.097525 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-model-cache\") pod \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " Sep 07 13:19:52.097619 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.097573 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp48g\" (UniqueName: \"kubernetes.io/projected/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-kube-api-access-gp48g\") pod \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " Sep 07 13:19:52.097864 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.097698 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-home" (OuterVolumeSpecName: "home") pod "2355b4b5-941c-4d2d-ab1a-57ffe05bc789" (UID: "2355b4b5-941c-4d2d-ab1a-57ffe05bc789"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:19:52.097864 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.097795 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-model-cache" (OuterVolumeSpecName: "model-cache") pod "2355b4b5-941c-4d2d-ab1a-57ffe05bc789" (UID: "2355b4b5-941c-4d2d-ab1a-57ffe05bc789"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:19:52.097864 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.097820 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tmp-dir\") pod \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\" (UID: \"2355b4b5-941c-4d2d-ab1a-57ffe05bc789\") " Sep 07 13:19:52.098093 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.098068 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2355b4b5-941c-4d2d-ab1a-57ffe05bc789" (UID: "2355b4b5-941c-4d2d-ab1a-57ffe05bc789"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:19:52.098200 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.098187 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-model-cache\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:19:52.098246 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.098204 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tmp-dir\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:19:52.098246 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.098212 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-home\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:19:52.099729 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.099710 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-dshm" (OuterVolumeSpecName: "dshm") pod "2355b4b5-941c-4d2d-ab1a-57ffe05bc789" (UID: "2355b4b5-941c-4d2d-ab1a-57ffe05bc789"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:19:52.100034 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.100006 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2355b4b5-941c-4d2d-ab1a-57ffe05bc789" (UID: "2355b4b5-941c-4d2d-ab1a-57ffe05bc789"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:19:52.100034 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.100008 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-kube-api-access-gp48g" (OuterVolumeSpecName: "kube-api-access-gp48g") pod "2355b4b5-941c-4d2d-ab1a-57ffe05bc789" (UID: "2355b4b5-941c-4d2d-ab1a-57ffe05bc789"). InnerVolumeSpecName "kube-api-access-gp48g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:19:52.199528 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.199498 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp48g\" (UniqueName: \"kubernetes.io/projected/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-kube-api-access-gp48g\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:19:52.199528 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.199523 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-dshm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:19:52.199528 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.199533 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2355b4b5-941c-4d2d-ab1a-57ffe05bc789-tls-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:19:52.575275 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.575239 2565 generic.go:356] "Generic (PLEG): container finished" podID="2355b4b5-941c-4d2d-ab1a-57ffe05bc789" containerID="65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb" exitCode=0 Sep 07 13:19:52.575275 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.575275 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" Sep 07 13:19:52.575476 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.575323 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" event={"ID":"2355b4b5-941c-4d2d-ab1a-57ffe05bc789","Type":"ContainerDied","Data":"65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb"} Sep 07 13:19:52.575476 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.575363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz" event={"ID":"2355b4b5-941c-4d2d-ab1a-57ffe05bc789","Type":"ContainerDied","Data":"d031719d27576d82d2c4a0ba313c0aa258af121561438225995fd9dfc418e94b"} Sep 07 13:19:52.575476 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.575384 2565 scope.go:117] "RemoveContainer" containerID="65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb" Sep 07 13:19:52.585016 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.584998 2565 scope.go:117] "RemoveContainer" containerID="65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb" Sep 07 13:19:52.585276 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:19:52.585258 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb\": container with ID starting with 65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb not found: ID does not exist" containerID="65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb" Sep 07 13:19:52.585325 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.585284 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb"} err="failed to get container status \"65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb\": rpc error: code = NotFound desc = could not find container \"65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb\": container with ID starting with 65ed806a2d36283747e9befda7db8c2ed28e82188cf089d698d19a3c32f91aeb not found: ID does not exist" Sep 07 13:19:52.598323 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.598300 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz"] Sep 07 13:19:52.602835 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.602811 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-067a48/noten-model-067a48-kserve-5749c6d687-kxnmz"] Sep 07 13:19:52.826434 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:52.826363 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2355b4b5-941c-4d2d-ab1a-57ffe05bc789" path="/var/lib/kubelet/pods/2355b4b5-941c-4d2d-ab1a-57ffe05bc789/volumes" Sep 07 13:19:57.547648 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:19:57.547583 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:20:02.559877 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.559842 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb"] Sep 07 13:20:02.560274 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.560252 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2355b4b5-941c-4d2d-ab1a-57ffe05bc789" containerName="main" Sep 07 13:20:02.560274 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.560273 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2355b4b5-941c-4d2d-ab1a-57ffe05bc789" containerName="main" Sep 07 13:20:02.560361 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.560337 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2355b4b5-941c-4d2d-ab1a-57ffe05bc789" containerName="main" Sep 07 13:20:02.565007 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.564992 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.568133 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.568103 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5910bb\"/\"default-dockercfg-984gj\"" Sep 07 13:20:02.569084 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.568892 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5910bb\"/\"kube-root-ca.crt\"" Sep 07 13:20:02.569342 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.569316 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5910bb\"/\"auth-test-model-5910bb-kserve-self-signed-certs\"" Sep 07 13:20:02.576245 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.570638 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5910bb\"/\"openshift-service-ca.crt\"" Sep 07 13:20:02.577202 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.577174 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb"] Sep 07 13:20:02.694302 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.694268 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68037297-ae8d-43a5-939c-f1354363ab9e-tls-certs\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.694482 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.694323 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-home\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.694482 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.694402 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-dshm\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.694482 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.694443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-model-cache\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.694706 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.694532 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-tmp-dir\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.694706 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.694572 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmz8m\" (UniqueName: \"kubernetes.io/projected/68037297-ae8d-43a5-939c-f1354363ab9e-kube-api-access-hmz8m\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.795627 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.795581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-model-cache\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.795757 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.795646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-tmp-dir\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.795757 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.795671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmz8m\" (UniqueName: \"kubernetes.io/projected/68037297-ae8d-43a5-939c-f1354363ab9e-kube-api-access-hmz8m\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.795757 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.795702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68037297-ae8d-43a5-939c-f1354363ab9e-tls-certs\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.795757 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.795744 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-home\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.795976 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.795782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-dshm\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.796059 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.796038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-model-cache\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.796116 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.796057 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-tmp-dir\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.796158 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.796122 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-home\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.797927 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.797906 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-dshm\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.798081 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.798064 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68037297-ae8d-43a5-939c-f1354363ab9e-tls-certs\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.807973 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.807953 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmz8m\" (UniqueName: \"kubernetes.io/projected/68037297-ae8d-43a5-939c-f1354363ab9e-kube-api-access-hmz8m\") pod \"auth-test-model-5910bb-kserve-59758bf6f5-2lsxb\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:02.879556 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:02.879497 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:03.156275 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:03.156129 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb"] Sep 07 13:20:03.158924 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:20:03.158898 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68037297_ae8d_43a5_939c_f1354363ab9e.slice/crio-44090a017e9674e74c15e04292e1e798061755fb2227d7d2f84a5dd72eb9acb1 WatchSource:0}: Error finding container 44090a017e9674e74c15e04292e1e798061755fb2227d7d2f84a5dd72eb9acb1: Status 404 returned error can't find the container with id 44090a017e9674e74c15e04292e1e798061755fb2227d7d2f84a5dd72eb9acb1 Sep 07 13:20:03.631204 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:03.631115 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" event={"ID":"68037297-ae8d-43a5-939c-f1354363ab9e","Type":"ContainerStarted","Data":"f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373"} Sep 07 13:20:03.631204 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:03.631158 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" event={"ID":"68037297-ae8d-43a5-939c-f1354363ab9e","Type":"ContainerStarted","Data":"44090a017e9674e74c15e04292e1e798061755fb2227d7d2f84a5dd72eb9acb1"} Sep 07 13:20:03.667362 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:03.667310 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" podStartSLOduration=1.667294349 podStartE2EDuration="1.667294349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:20:03.666263947 +0000 UTC m=+1225.346649717" watchObservedRunningTime="2026-09-07 13:20:03.667294349 +0000 UTC m=+1225.347680171" Sep 07 13:20:12.879818 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:12.879783 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:12.879818 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:12.879822 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:12.892339 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:12.892313 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:13.692225 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:13.692197 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:20:14.668606 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.668560 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd"] Sep 07 13:20:14.672223 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.672203 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:20:14.675102 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.675074 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-8e7ddc\"" Sep 07 13:20:14.675210 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.675188 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-8e7ddc-dockercfg-65679\"" Sep 07 13:20:14.684534 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.684511 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd"] Sep 07 13:20:14.702323 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.702296 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd\" (UID: \"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:20:14.702447 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.702333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgsjl\" (UniqueName: \"kubernetes.io/projected/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-kube-api-access-hgsjl\") pod \"payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd\" (UID: \"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:20:14.753741 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.753715 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq"] Sep 07 13:20:14.757182 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.757167 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:20:14.770397 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.770377 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq"] Sep 07 13:20:14.803448 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.803419 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxhbl\" (UniqueName: \"kubernetes.io/projected/d20267fc-bec6-4bf4-8bfc-62388088fac7-kube-api-access-pxhbl\") pod \"payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq\" (UID: \"d20267fc-bec6-4bf4-8bfc-62388088fac7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:20:14.803551 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.803465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd\" (UID: \"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:20:14.803551 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.803523 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgsjl\" (UniqueName: \"kubernetes.io/projected/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-kube-api-access-hgsjl\") pod \"payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd\" (UID: \"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:20:14.803654 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.803568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20267fc-bec6-4bf4-8bfc-62388088fac7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq\" (UID: \"d20267fc-bec6-4bf4-8bfc-62388088fac7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:20:14.804185 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.804167 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd\" (UID: \"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:20:14.814309 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.814291 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgsjl\" (UniqueName: \"kubernetes.io/projected/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-kube-api-access-hgsjl\") pod \"payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd\" (UID: \"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:20:14.904893 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.904865 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20267fc-bec6-4bf4-8bfc-62388088fac7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq\" (UID: \"d20267fc-bec6-4bf4-8bfc-62388088fac7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:20:14.905057 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.904942 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxhbl\" (UniqueName: \"kubernetes.io/projected/d20267fc-bec6-4bf4-8bfc-62388088fac7-kube-api-access-pxhbl\") pod \"payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq\" (UID: \"d20267fc-bec6-4bf4-8bfc-62388088fac7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:20:14.905581 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.905558 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20267fc-bec6-4bf4-8bfc-62388088fac7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq\" (UID: \"d20267fc-bec6-4bf4-8bfc-62388088fac7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:20:14.914241 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.914219 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxhbl\" (UniqueName: \"kubernetes.io/projected/d20267fc-bec6-4bf4-8bfc-62388088fac7-kube-api-access-pxhbl\") pod \"payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq\" (UID: \"d20267fc-bec6-4bf4-8bfc-62388088fac7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:20:14.983058 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:14.982981 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:20:15.067885 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:15.067789 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:20:15.260992 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:15.260968 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd"] Sep 07 13:20:15.262905 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:20:15.262879 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3786fb7e_0ac3_4873_95d1_6b4fb7bab98d.slice/crio-181ac2bb03b98829743d75c8c8c77c985e3713e41c70bcea496492dc08c2ec04 WatchSource:0}: Error finding container 181ac2bb03b98829743d75c8c8c77c985e3713e41c70bcea496492dc08c2ec04: Status 404 returned error can't find the container with id 181ac2bb03b98829743d75c8c8c77c985e3713e41c70bcea496492dc08c2ec04 Sep 07 13:20:15.357286 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:15.357261 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq"] Sep 07 13:20:15.359530 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:20:15.359506 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20267fc_bec6_4bf4_8bfc_62388088fac7.slice/crio-76f43cfe680f4f4733132d56ddb340546847326f21091c13648d335eec953e6c WatchSource:0}: Error finding container 76f43cfe680f4f4733132d56ddb340546847326f21091c13648d335eec953e6c: Status 404 returned error can't find the container with id 76f43cfe680f4f4733132d56ddb340546847326f21091c13648d335eec953e6c Sep 07 13:20:15.694041 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:15.694005 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" event={"ID":"d20267fc-bec6-4bf4-8bfc-62388088fac7","Type":"ContainerStarted","Data":"a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba"} Sep 07 13:20:15.694506 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:15.694066 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" event={"ID":"d20267fc-bec6-4bf4-8bfc-62388088fac7","Type":"ContainerStarted","Data":"76f43cfe680f4f4733132d56ddb340546847326f21091c13648d335eec953e6c"} Sep 07 13:20:15.694506 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:15.694193 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:20:15.695751 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:15.695723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" event={"ID":"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d","Type":"ContainerStarted","Data":"1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf"} Sep 07 13:20:15.695889 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:15.695768 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" event={"ID":"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d","Type":"ContainerStarted","Data":"181ac2bb03b98829743d75c8c8c77c985e3713e41c70bcea496492dc08c2ec04"} Sep 07 13:20:15.695889 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:15.695811 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:20:15.718777 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:15.718722 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" podStartSLOduration=1.7187041650000001 podStartE2EDuration="1.718704165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:20:15.716416692 +0000 UTC m=+1237.396802464" watchObservedRunningTime="2026-09-07 13:20:15.718704165 +0000 UTC m=+1237.399089936" Sep 07 13:20:15.742619 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:15.742566 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" podStartSLOduration=1.742553772 podStartE2EDuration="1.742553772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:20:15.740618194 +0000 UTC m=+1237.421003961" watchObservedRunningTime="2026-09-07 13:20:15.742553772 +0000 UTC m=+1237.422939686" Sep 07 13:20:26.703070 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:26.703040 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:20:26.705480 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:26.703415 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:20:38.985334 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:20:38.985262 2565 scope.go:117] "RemoveContainer" containerID="788a13050294f20b3bab1a6ba9ae46ea3aa9890f9812f8caf36c6aaaa651121f" Sep 07 13:21:01.688414 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:01.688376 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6"] Sep 07 13:21:01.693726 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:01.693705 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:21:01.696590 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:01.696567 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ede2050e\"" Sep 07 13:21:01.708079 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:01.708054 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6"] Sep 07 13:21:01.739045 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:01.739010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c577cf34-ab7d-4d27-91ae-92482d330b12-plugins-config-volume\") pod \"payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6\" (UID: \"c577cf34-ab7d-4d27-91ae-92482d330b12\") " pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:21:01.739214 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:01.739075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6xg\" (UniqueName: \"kubernetes.io/projected/c577cf34-ab7d-4d27-91ae-92482d330b12-kube-api-access-6q6xg\") pod \"payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6\" (UID: \"c577cf34-ab7d-4d27-91ae-92482d330b12\") " pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:21:01.839846 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:01.839809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c577cf34-ab7d-4d27-91ae-92482d330b12-plugins-config-volume\") pod \"payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6\" (UID: \"c577cf34-ab7d-4d27-91ae-92482d330b12\") " pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:21:01.840039 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:01.839866 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6xg\" (UniqueName: \"kubernetes.io/projected/c577cf34-ab7d-4d27-91ae-92482d330b12-kube-api-access-6q6xg\") pod \"payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6\" (UID: \"c577cf34-ab7d-4d27-91ae-92482d330b12\") " pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:21:01.840627 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:01.840577 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c577cf34-ab7d-4d27-91ae-92482d330b12-plugins-config-volume\") pod \"payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6\" (UID: \"c577cf34-ab7d-4d27-91ae-92482d330b12\") " pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:21:01.849167 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:01.849137 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6xg\" (UniqueName: \"kubernetes.io/projected/c577cf34-ab7d-4d27-91ae-92482d330b12-kube-api-access-6q6xg\") pod \"payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6\" (UID: \"c577cf34-ab7d-4d27-91ae-92482d330b12\") " pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:21:02.004270 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:02.004184 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:21:02.279702 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:02.279677 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6"] Sep 07 13:21:02.281727 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:21:02.281699 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc577cf34_ab7d_4d27_91ae_92482d330b12.slice/crio-63e6187f00858145c7454573c3c1539e6a7d84b1e6ed1a18f120da96308129d3 WatchSource:0}: Error finding container 63e6187f00858145c7454573c3c1539e6a7d84b1e6ed1a18f120da96308129d3: Status 404 returned error can't find the container with id 63e6187f00858145c7454573c3c1539e6a7d84b1e6ed1a18f120da96308129d3 Sep 07 13:21:02.945832 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:02.945795 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" event={"ID":"c577cf34-ab7d-4d27-91ae-92482d330b12","Type":"ContainerStarted","Data":"624464deb84f9b01c4b2b8b3c6319ec6c7cb811f93e008ac36794bfff626ed51"} Sep 07 13:21:02.946193 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:02.945859 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" event={"ID":"c577cf34-ab7d-4d27-91ae-92482d330b12","Type":"ContainerStarted","Data":"63e6187f00858145c7454573c3c1539e6a7d84b1e6ed1a18f120da96308129d3"} Sep 07 13:21:02.946193 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:02.945921 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:21:02.987275 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:02.987225 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" podStartSLOduration=1.9872101500000001 podStartE2EDuration="1.98721015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:21:02.985517996 +0000 UTC m=+1284.665903765" watchObservedRunningTime="2026-09-07 13:21:02.98721015 +0000 UTC m=+1284.667595920" Sep 07 13:21:13.953584 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:13.953551 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:21:35.829275 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.829245 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657"] Sep 07 13:21:35.832734 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.832717 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.836988 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.836965 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ede2050e\"/\"kube-root-ca.crt\"" Sep 07 13:21:35.837479 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.837463 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ede2050e\"/\"default-dockercfg-zmwxj\"" Sep 07 13:21:35.837574 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.837484 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ede2050e\"/\"openshift-service-ca.crt\"" Sep 07 13:21:35.837681 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.837653 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ede2050e\"/\"e2e-lifecycle-model-ede2050e-kserve-self-signed-certs\"" Sep 07 13:21:35.846101 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.846077 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657"] Sep 07 13:21:35.858686 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.858662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-tmp-dir\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.858777 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.858741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm5jv\" (UniqueName: \"kubernetes.io/projected/5a665bd0-3c88-4be4-8347-a46960382e45-kube-api-access-pm5jv\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.858777 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.858758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-model-cache\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.858868 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.858850 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a665bd0-3c88-4be4-8347-a46960382e45-tls-certs\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.858922 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.858895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-dshm\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.859128 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.859060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-home\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.960203 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.960171 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm5jv\" (UniqueName: \"kubernetes.io/projected/5a665bd0-3c88-4be4-8347-a46960382e45-kube-api-access-pm5jv\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.960203 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.960203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-model-cache\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.960446 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.960251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a665bd0-3c88-4be4-8347-a46960382e45-tls-certs\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.960446 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.960287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-dshm\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.960446 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.960309 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-home\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.960446 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.960426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-tmp-dir\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.960702 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.960652 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-model-cache\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.960743 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.960722 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-home\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.960783 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.960768 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-tmp-dir\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.962435 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.962416 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-dshm\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.962755 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.962740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a665bd0-3c88-4be4-8347-a46960382e45-tls-certs\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:35.969457 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:35.969437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm5jv\" (UniqueName: \"kubernetes.io/projected/5a665bd0-3c88-4be4-8347-a46960382e45-kube-api-access-pm5jv\") pod \"e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:36.142394 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:36.142311 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:36.425415 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:36.425354 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657"] Sep 07 13:21:36.428068 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:21:36.428037 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a665bd0_3c88_4be4_8347_a46960382e45.slice/crio-2c9519bbc9f3fb3ad3d4e87971cd8b1b8c68f3e0fa2b0c8534b59a21664c3a31 WatchSource:0}: Error finding container 2c9519bbc9f3fb3ad3d4e87971cd8b1b8c68f3e0fa2b0c8534b59a21664c3a31: Status 404 returned error can't find the container with id 2c9519bbc9f3fb3ad3d4e87971cd8b1b8c68f3e0fa2b0c8534b59a21664c3a31 Sep 07 13:21:37.121946 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:37.121905 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" event={"ID":"5a665bd0-3c88-4be4-8347-a46960382e45","Type":"ContainerStarted","Data":"5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e"} Sep 07 13:21:37.121946 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:37.121949 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" event={"ID":"5a665bd0-3c88-4be4-8347-a46960382e45","Type":"ContainerStarted","Data":"2c9519bbc9f3fb3ad3d4e87971cd8b1b8c68f3e0fa2b0c8534b59a21664c3a31"} Sep 07 13:21:39.009554 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:39.009521 2565 scope.go:117] "RemoveContainer" containerID="f3fdd867264dacc546fd52812a5c23475f942d9e444836653fc945bf19775608" Sep 07 13:21:39.017921 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:39.017902 2565 scope.go:117] "RemoveContainer" containerID="d826388d8fc3cee4e783af6943259c754f032676750eddb3b6826e2fdd3589d6" Sep 07 13:21:46.142833 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:46.142801 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:46.143263 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:46.142854 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:46.155570 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:46.155547 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:46.178455 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:46.178431 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:21:46.181416 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:21:46.181370 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" podStartSLOduration=11.181356676 podStartE2EDuration="11.181356676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:21:37.164674824 +0000 UTC m=+1318.845060594" watchObservedRunningTime="2026-09-07 13:21:46.181356676 +0000 UTC m=+1327.861742523" Sep 07 13:22:25.189608 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:25.189558 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6"] Sep 07 13:22:25.190130 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:25.189820 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" podUID="c577cf34-ab7d-4d27-91ae-92482d330b12" containerName="payload-processing" containerID="cri-o://624464deb84f9b01c4b2b8b3c6319ec6c7cb811f93e008ac36794bfff626ed51" gracePeriod=30 Sep 07 13:22:25.369066 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:25.369035 2565 generic.go:356] "Generic (PLEG): container finished" podID="c577cf34-ab7d-4d27-91ae-92482d330b12" containerID="624464deb84f9b01c4b2b8b3c6319ec6c7cb811f93e008ac36794bfff626ed51" exitCode=0 Sep 07 13:22:25.369223 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:25.369086 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" event={"ID":"c577cf34-ab7d-4d27-91ae-92482d330b12","Type":"ContainerDied","Data":"624464deb84f9b01c4b2b8b3c6319ec6c7cb811f93e008ac36794bfff626ed51"} Sep 07 13:22:25.458368 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:25.458345 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:22:25.541318 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:25.541284 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c577cf34-ab7d-4d27-91ae-92482d330b12-plugins-config-volume\") pod \"c577cf34-ab7d-4d27-91ae-92482d330b12\" (UID: \"c577cf34-ab7d-4d27-91ae-92482d330b12\") " Sep 07 13:22:25.541462 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:25.541410 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q6xg\" (UniqueName: \"kubernetes.io/projected/c577cf34-ab7d-4d27-91ae-92482d330b12-kube-api-access-6q6xg\") pod \"c577cf34-ab7d-4d27-91ae-92482d330b12\" (UID: \"c577cf34-ab7d-4d27-91ae-92482d330b12\") " Sep 07 13:22:25.541741 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:25.541720 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c577cf34-ab7d-4d27-91ae-92482d330b12-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c577cf34-ab7d-4d27-91ae-92482d330b12" (UID: "c577cf34-ab7d-4d27-91ae-92482d330b12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:22:25.541814 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:25.541798 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c577cf34-ab7d-4d27-91ae-92482d330b12-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:25.543509 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:25.543483 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c577cf34-ab7d-4d27-91ae-92482d330b12-kube-api-access-6q6xg" (OuterVolumeSpecName: "kube-api-access-6q6xg") pod "c577cf34-ab7d-4d27-91ae-92482d330b12" (UID: "c577cf34-ab7d-4d27-91ae-92482d330b12"). InnerVolumeSpecName "kube-api-access-6q6xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:22:25.643086 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:25.643056 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6xg\" (UniqueName: \"kubernetes.io/projected/c577cf34-ab7d-4d27-91ae-92482d330b12-kube-api-access-6q6xg\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:26.382447 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:26.382409 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" event={"ID":"c577cf34-ab7d-4d27-91ae-92482d330b12","Type":"ContainerDied","Data":"63e6187f00858145c7454573c3c1539e6a7d84b1e6ed1a18f120da96308129d3"} Sep 07 13:22:26.382447 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:26.382452 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6" Sep 07 13:22:26.382965 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:26.382497 2565 scope.go:117] "RemoveContainer" containerID="624464deb84f9b01c4b2b8b3c6319ec6c7cb811f93e008ac36794bfff626ed51" Sep 07 13:22:26.409356 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:26.409328 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6"] Sep 07 13:22:26.413981 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:26.413961 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ede2050e-85475f958f-9wlc6"] Sep 07 13:22:26.827192 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:26.827159 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c577cf34-ab7d-4d27-91ae-92482d330b12" path="/var/lib/kubelet/pods/c577cf34-ab7d-4d27-91ae-92482d330b12/volumes" Sep 07 13:22:32.871895 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:32.871858 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb"] Sep 07 13:22:33.717375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:33.717336 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657"] Sep 07 13:22:47.890987 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:47.890948 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" podUID="68037297-ae8d-43a5-939c-f1354363ab9e" containerName="main" containerID="cri-o://f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373" gracePeriod=45 Sep 07 13:22:48.141895 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.141836 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:22:48.157301 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.157274 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-tmp-dir\") pod \"68037297-ae8d-43a5-939c-f1354363ab9e\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " Sep 07 13:22:48.157424 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.157343 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmz8m\" (UniqueName: \"kubernetes.io/projected/68037297-ae8d-43a5-939c-f1354363ab9e-kube-api-access-hmz8m\") pod \"68037297-ae8d-43a5-939c-f1354363ab9e\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " Sep 07 13:22:48.157424 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.157385 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68037297-ae8d-43a5-939c-f1354363ab9e-tls-certs\") pod \"68037297-ae8d-43a5-939c-f1354363ab9e\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " Sep 07 13:22:48.157530 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.157436 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-dshm\") pod \"68037297-ae8d-43a5-939c-f1354363ab9e\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " Sep 07 13:22:48.157530 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.157460 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-model-cache\") pod \"68037297-ae8d-43a5-939c-f1354363ab9e\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " Sep 07 13:22:48.157530 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.157508 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-home\") pod \"68037297-ae8d-43a5-939c-f1354363ab9e\" (UID: \"68037297-ae8d-43a5-939c-f1354363ab9e\") " Sep 07 13:22:48.157736 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.157554 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68037297-ae8d-43a5-939c-f1354363ab9e" (UID: "68037297-ae8d-43a5-939c-f1354363ab9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:22:48.157858 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.157828 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-tmp-dir\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:48.157987 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.157859 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-home" (OuterVolumeSpecName: "home") pod "68037297-ae8d-43a5-939c-f1354363ab9e" (UID: "68037297-ae8d-43a5-939c-f1354363ab9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:22:48.158049 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.157995 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "68037297-ae8d-43a5-939c-f1354363ab9e" (UID: "68037297-ae8d-43a5-939c-f1354363ab9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:22:48.160062 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.160031 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-dshm" (OuterVolumeSpecName: "dshm") pod "68037297-ae8d-43a5-939c-f1354363ab9e" (UID: "68037297-ae8d-43a5-939c-f1354363ab9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:22:48.160164 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.160086 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68037297-ae8d-43a5-939c-f1354363ab9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68037297-ae8d-43a5-939c-f1354363ab9e" (UID: "68037297-ae8d-43a5-939c-f1354363ab9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:22:48.160210 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.160167 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68037297-ae8d-43a5-939c-f1354363ab9e-kube-api-access-hmz8m" (OuterVolumeSpecName: "kube-api-access-hmz8m") pod "68037297-ae8d-43a5-939c-f1354363ab9e" (UID: "68037297-ae8d-43a5-939c-f1354363ab9e"). InnerVolumeSpecName "kube-api-access-hmz8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:22:48.259196 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.259173 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmz8m\" (UniqueName: \"kubernetes.io/projected/68037297-ae8d-43a5-939c-f1354363ab9e-kube-api-access-hmz8m\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:48.259196 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.259198 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68037297-ae8d-43a5-939c-f1354363ab9e-tls-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:48.259356 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.259208 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-dshm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:48.259356 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.259218 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-model-cache\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:48.259356 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.259225 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68037297-ae8d-43a5-939c-f1354363ab9e-home\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:48.497909 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.497877 2565 generic.go:356] "Generic (PLEG): container finished" podID="68037297-ae8d-43a5-939c-f1354363ab9e" containerID="f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373" exitCode=0 Sep 07 13:22:48.498115 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.497917 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" Sep 07 13:22:48.498115 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.497963 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" event={"ID":"68037297-ae8d-43a5-939c-f1354363ab9e","Type":"ContainerDied","Data":"f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373"} Sep 07 13:22:48.498115 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.498000 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb" event={"ID":"68037297-ae8d-43a5-939c-f1354363ab9e","Type":"ContainerDied","Data":"44090a017e9674e74c15e04292e1e798061755fb2227d7d2f84a5dd72eb9acb1"} Sep 07 13:22:48.498115 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.498024 2565 scope.go:117] "RemoveContainer" containerID="f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373" Sep 07 13:22:48.507494 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.507476 2565 scope.go:117] "RemoveContainer" containerID="f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373" Sep 07 13:22:48.507750 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:22:48.507733 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373\": container with ID starting with f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373 not found: ID does not exist" containerID="f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373" Sep 07 13:22:48.507820 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.507758 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373"} err="failed to get container status \"f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373\": rpc error: code = NotFound desc = could not find container \"f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373\": container with ID starting with f1e36d7830a59ea53b7ba7072c65de0326d75767258c6f2a85f6bb9013cf0373 not found: ID does not exist" Sep 07 13:22:48.526143 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.526121 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb"] Sep 07 13:22:48.537817 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.537795 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/auth-test-model-5910bb-kserve-59758bf6f5-2lsxb"] Sep 07 13:22:48.737409 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.737373 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" podUID="5a665bd0-3c88-4be4-8347-a46960382e45" containerName="main" containerID="cri-o://5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e" gracePeriod=45 Sep 07 13:22:48.831738 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.831670 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68037297-ae8d-43a5-939c-f1354363ab9e" path="/var/lib/kubelet/pods/68037297-ae8d-43a5-939c-f1354363ab9e/volumes" Sep 07 13:22:48.989905 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:48.989883 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:22:49.065243 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065209 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-home\") pod \"5a665bd0-3c88-4be4-8347-a46960382e45\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " Sep 07 13:22:49.065243 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065279 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-tmp-dir\") pod \"5a665bd0-3c88-4be4-8347-a46960382e45\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " Sep 07 13:22:49.065517 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065305 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a665bd0-3c88-4be4-8347-a46960382e45-tls-certs\") pod \"5a665bd0-3c88-4be4-8347-a46960382e45\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " Sep 07 13:22:49.065517 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065324 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-dshm\") pod \"5a665bd0-3c88-4be4-8347-a46960382e45\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " Sep 07 13:22:49.065517 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065352 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm5jv\" (UniqueName: \"kubernetes.io/projected/5a665bd0-3c88-4be4-8347-a46960382e45-kube-api-access-pm5jv\") pod \"5a665bd0-3c88-4be4-8347-a46960382e45\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " Sep 07 13:22:49.065517 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065417 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-model-cache\") pod \"5a665bd0-3c88-4be4-8347-a46960382e45\" (UID: \"5a665bd0-3c88-4be4-8347-a46960382e45\") " Sep 07 13:22:49.065517 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065444 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-home" (OuterVolumeSpecName: "home") pod "5a665bd0-3c88-4be4-8347-a46960382e45" (UID: "5a665bd0-3c88-4be4-8347-a46960382e45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:22:49.065813 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065584 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a665bd0-3c88-4be4-8347-a46960382e45" (UID: "5a665bd0-3c88-4be4-8347-a46960382e45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:22:49.065813 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065721 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a665bd0-3c88-4be4-8347-a46960382e45" (UID: "5a665bd0-3c88-4be4-8347-a46960382e45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:22:49.066000 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065835 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-model-cache\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:49.066000 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065850 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-home\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:49.066000 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.065858 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-tmp-dir\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:49.067490 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.067457 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-dshm" (OuterVolumeSpecName: "dshm") pod "5a665bd0-3c88-4be4-8347-a46960382e45" (UID: "5a665bd0-3c88-4be4-8347-a46960382e45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:22:49.067637 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.067506 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a665bd0-3c88-4be4-8347-a46960382e45-kube-api-access-pm5jv" (OuterVolumeSpecName: "kube-api-access-pm5jv") pod "5a665bd0-3c88-4be4-8347-a46960382e45" (UID: "5a665bd0-3c88-4be4-8347-a46960382e45"). InnerVolumeSpecName "kube-api-access-pm5jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:22:49.067637 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.067565 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a665bd0-3c88-4be4-8347-a46960382e45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a665bd0-3c88-4be4-8347-a46960382e45" (UID: "5a665bd0-3c88-4be4-8347-a46960382e45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:22:49.166431 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.166403 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a665bd0-3c88-4be4-8347-a46960382e45-tls-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:49.166431 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.166428 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a665bd0-3c88-4be4-8347-a46960382e45-dshm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:49.166431 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.166436 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm5jv\" (UniqueName: \"kubernetes.io/projected/5a665bd0-3c88-4be4-8347-a46960382e45-kube-api-access-pm5jv\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:22:49.504457 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.504373 2565 generic.go:356] "Generic (PLEG): container finished" podID="5a665bd0-3c88-4be4-8347-a46960382e45" containerID="5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e" exitCode=0 Sep 07 13:22:49.504457 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.504417 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" Sep 07 13:22:49.504706 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.504459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" event={"ID":"5a665bd0-3c88-4be4-8347-a46960382e45","Type":"ContainerDied","Data":"5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e"} Sep 07 13:22:49.504706 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.504501 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657" event={"ID":"5a665bd0-3c88-4be4-8347-a46960382e45","Type":"ContainerDied","Data":"2c9519bbc9f3fb3ad3d4e87971cd8b1b8c68f3e0fa2b0c8534b59a21664c3a31"} Sep 07 13:22:49.504706 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.504523 2565 scope.go:117] "RemoveContainer" containerID="5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e" Sep 07 13:22:49.513735 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.513715 2565 scope.go:117] "RemoveContainer" containerID="5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e" Sep 07 13:22:49.513985 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:22:49.513969 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e\": container with ID starting with 5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e not found: ID does not exist" containerID="5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e" Sep 07 13:22:49.514039 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.513993 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e"} err="failed to get container status \"5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e\": rpc error: code = NotFound desc = could not find container \"5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e\": container with ID starting with 5a30f15d463c18dbd26635ba47614838f83d62805224d12d6f09ba2785c7ec9e not found: ID does not exist" Sep 07 13:22:49.543815 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.543792 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657"] Sep 07 13:22:49.553609 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:49.553573 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ede2050e/e2e-lifecycle-model-ede2050e-kserve-7bd47575bc-75657"] Sep 07 13:22:50.827281 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:22:50.827249 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a665bd0-3c88-4be4-8347-a46960382e45" path="/var/lib/kubelet/pods/5a665bd0-3c88-4be4-8347-a46960382e45/volumes" Sep 07 13:23:02.242169 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.242136 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96"] Sep 07 13:23:02.242720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.242702 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a665bd0-3c88-4be4-8347-a46960382e45" containerName="main" Sep 07 13:23:02.242720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.242722 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a665bd0-3c88-4be4-8347-a46960382e45" containerName="main" Sep 07 13:23:02.242830 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.242737 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68037297-ae8d-43a5-939c-f1354363ab9e" containerName="main" Sep 07 13:23:02.242830 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.242743 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="68037297-ae8d-43a5-939c-f1354363ab9e" containerName="main" Sep 07 13:23:02.242830 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.242756 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c577cf34-ab7d-4d27-91ae-92482d330b12" containerName="payload-processing" Sep 07 13:23:02.242830 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.242765 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c577cf34-ab7d-4d27-91ae-92482d330b12" containerName="payload-processing" Sep 07 13:23:02.242963 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.242846 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c577cf34-ab7d-4d27-91ae-92482d330b12" containerName="payload-processing" Sep 07 13:23:02.242963 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.242859 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="68037297-ae8d-43a5-939c-f1354363ab9e" containerName="main" Sep 07 13:23:02.242963 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.242866 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a665bd0-3c88-4be4-8347-a46960382e45" containerName="main" Sep 07 13:23:02.247423 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.247404 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.250569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.250548 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"sub-test-model-c7c371-kserve-self-signed-certs\"" Sep 07 13:23:02.251295 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.251273 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"kube-root-ca.crt\"" Sep 07 13:23:02.251402 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.251325 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"default-dockercfg-gc9sj\"" Sep 07 13:23:02.251402 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.251345 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c7c371\"/\"openshift-service-ca.crt\"" Sep 07 13:23:02.259808 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.259787 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96"] Sep 07 13:23:02.381315 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.381281 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-tmp-dir\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.381315 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.381317 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph8j8\" (UniqueName: \"kubernetes.io/projected/dd508ec6-6de8-432f-8400-9987187af2e1-kube-api-access-ph8j8\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.381527 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.381344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-dshm\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.381527 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.381404 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd508ec6-6de8-432f-8400-9987187af2e1-tls-certs\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.381527 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.381431 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-model-cache\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.381674 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.381526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-home\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.482520 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.482486 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-tmp-dir\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.482520 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.482526 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph8j8\" (UniqueName: \"kubernetes.io/projected/dd508ec6-6de8-432f-8400-9987187af2e1-kube-api-access-ph8j8\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.482802 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.482548 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-dshm\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.482802 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.482576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd508ec6-6de8-432f-8400-9987187af2e1-tls-certs\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.482802 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.482631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-model-cache\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.482802 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.482751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-home\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.483022 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.482936 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-tmp-dir\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.483022 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.482986 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-model-cache\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.483116 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.483040 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-home\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.484878 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.484858 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-dshm\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.485017 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.485001 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd508ec6-6de8-432f-8400-9987187af2e1-tls-certs\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.494497 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.494427 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph8j8\" (UniqueName: \"kubernetes.io/projected/dd508ec6-6de8-432f-8400-9987187af2e1-kube-api-access-ph8j8\") pod \"sub-test-model-c7c371-kserve-df859-cwj96\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.556435 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.556405 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:02.844618 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:02.844576 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96"] Sep 07 13:23:02.846723 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:23:02.846703 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd508ec6_6de8_432f_8400_9987187af2e1.slice/crio-24233663e5588576534d688ee3f5a48af00ca5829957286545cb836622d9495d WatchSource:0}: Error finding container 24233663e5588576534d688ee3f5a48af00ca5829957286545cb836622d9495d: Status 404 returned error can't find the container with id 24233663e5588576534d688ee3f5a48af00ca5829957286545cb836622d9495d Sep 07 13:23:03.579379 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:03.579343 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" event={"ID":"dd508ec6-6de8-432f-8400-9987187af2e1","Type":"ContainerStarted","Data":"6dce8c03df46dd85b35acac96d990075cae68ecbb7fe84e40fc07ed36ad10706"} Sep 07 13:23:03.579379 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:03.579382 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" event={"ID":"dd508ec6-6de8-432f-8400-9987187af2e1","Type":"ContainerStarted","Data":"24233663e5588576534d688ee3f5a48af00ca5829957286545cb836622d9495d"} Sep 07 13:23:03.605752 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:03.605701 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" podStartSLOduration=1.605686559 podStartE2EDuration="1.605686559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:23:03.603613949 +0000 UTC m=+1405.283999722" watchObservedRunningTime="2026-09-07 13:23:03.605686559 +0000 UTC m=+1405.286072329" Sep 07 13:23:12.557625 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:12.557574 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:12.558002 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:12.557645 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:12.569681 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:12.569657 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:12.636937 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:12.636913 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:23:51.752746 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:23:51.752713 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96"] Sep 07 13:24:02.862829 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.862790 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p"] Sep 07 13:24:02.867992 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.867970 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:02.871006 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.870987 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5910bb\"/\"default-dockercfg-984gj\"" Sep 07 13:24:02.871814 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.871795 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5910bb\"/\"kube-root-ca.crt\"" Sep 07 13:24:02.871921 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.871818 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5910bb\"/\"openshift-service-ca.crt\"" Sep 07 13:24:02.871977 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.871927 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5910bb\"/\"rate-test-model-5910bb-kserve-self-signed-certs\"" Sep 07 13:24:02.880320 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.880296 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p"] Sep 07 13:24:02.953009 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.952980 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tls-certs\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:02.953166 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.953041 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b59z\" (UniqueName: \"kubernetes.io/projected/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-kube-api-access-5b59z\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:02.953166 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.953061 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-home\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:02.953166 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.953078 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-dshm\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:02.953166 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.953124 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tmp-dir\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:02.953166 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:02.953159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-model-cache\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.054450 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.054417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tls-certs\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.054637 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.054487 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b59z\" (UniqueName: \"kubernetes.io/projected/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-kube-api-access-5b59z\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.054637 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.054516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-home\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.054637 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.054541 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-dshm\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.054637 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.054571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tmp-dir\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.054637 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.054613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-model-cache\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.055017 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.054954 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-home\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.055112 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.055056 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tmp-dir\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.055175 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.055104 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-model-cache\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.056916 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.056889 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-dshm\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.057010 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.056996 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tls-certs\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.063866 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.063845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b59z\" (UniqueName: \"kubernetes.io/projected/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-kube-api-access-5b59z\") pod \"rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.178220 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.178194 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:03.454899 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.454868 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p"] Sep 07 13:24:03.460090 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:24:03.460064 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod542040c9_b85c_43b5_b6b2_c2daf66f3ac6.slice/crio-329d7217149c7d14eea7cc6378c5049f16060c177b502a7976f1f2d7e5bc6e2a WatchSource:0}: Error finding container 329d7217149c7d14eea7cc6378c5049f16060c177b502a7976f1f2d7e5bc6e2a: Status 404 returned error can't find the container with id 329d7217149c7d14eea7cc6378c5049f16060c177b502a7976f1f2d7e5bc6e2a Sep 07 13:24:03.882334 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.882238 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" event={"ID":"542040c9-b85c-43b5-b6b2-c2daf66f3ac6","Type":"ContainerStarted","Data":"7722cdd00eff64b9bf4e72dc9e310652db117394802d132f3492c5bb3aff0f63"} Sep 07 13:24:03.882334 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.882289 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" event={"ID":"542040c9-b85c-43b5-b6b2-c2daf66f3ac6","Type":"ContainerStarted","Data":"329d7217149c7d14eea7cc6378c5049f16060c177b502a7976f1f2d7e5bc6e2a"} Sep 07 13:24:03.903828 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:03.903780 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" podStartSLOduration=1.9037656360000001 podStartE2EDuration="1.903765636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:24:03.903206319 +0000 UTC m=+1465.583592090" watchObservedRunningTime="2026-09-07 13:24:03.903765636 +0000 UTC m=+1465.584151406" Sep 07 13:24:06.772222 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:06.772151 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" podUID="dd508ec6-6de8-432f-8400-9987187af2e1" containerName="main" containerID="cri-o://6dce8c03df46dd85b35acac96d990075cae68ecbb7fe84e40fc07ed36ad10706" gracePeriod=45 Sep 07 13:24:06.901568 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:06.901537 2565 generic.go:356] "Generic (PLEG): container finished" podID="dd508ec6-6de8-432f-8400-9987187af2e1" containerID="6dce8c03df46dd85b35acac96d990075cae68ecbb7fe84e40fc07ed36ad10706" exitCode=0 Sep 07 13:24:06.901733 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:06.901631 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" event={"ID":"dd508ec6-6de8-432f-8400-9987187af2e1","Type":"ContainerDied","Data":"6dce8c03df46dd85b35acac96d990075cae68ecbb7fe84e40fc07ed36ad10706"} Sep 07 13:24:07.031545 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.031466 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:24:07.091271 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.091243 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-tmp-dir\") pod \"dd508ec6-6de8-432f-8400-9987187af2e1\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " Sep 07 13:24:07.091464 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.091285 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-dshm\") pod \"dd508ec6-6de8-432f-8400-9987187af2e1\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " Sep 07 13:24:07.091464 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.091345 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph8j8\" (UniqueName: \"kubernetes.io/projected/dd508ec6-6de8-432f-8400-9987187af2e1-kube-api-access-ph8j8\") pod \"dd508ec6-6de8-432f-8400-9987187af2e1\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " Sep 07 13:24:07.091464 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.091361 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-model-cache\") pod \"dd508ec6-6de8-432f-8400-9987187af2e1\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " Sep 07 13:24:07.091464 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.091381 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd508ec6-6de8-432f-8400-9987187af2e1-tls-certs\") pod \"dd508ec6-6de8-432f-8400-9987187af2e1\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " Sep 07 13:24:07.091464 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.091405 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-home\") pod \"dd508ec6-6de8-432f-8400-9987187af2e1\" (UID: \"dd508ec6-6de8-432f-8400-9987187af2e1\") " Sep 07 13:24:07.091878 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.091520 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd508ec6-6de8-432f-8400-9987187af2e1" (UID: "dd508ec6-6de8-432f-8400-9987187af2e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:24:07.091878 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.091645 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd508ec6-6de8-432f-8400-9987187af2e1" (UID: "dd508ec6-6de8-432f-8400-9987187af2e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:24:07.091878 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.091829 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-home" (OuterVolumeSpecName: "home") pod "dd508ec6-6de8-432f-8400-9987187af2e1" (UID: "dd508ec6-6de8-432f-8400-9987187af2e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:24:07.092130 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.092111 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-model-cache\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:24:07.092184 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.092136 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-home\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:24:07.092184 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.092150 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-tmp-dir\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:24:07.094357 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.094316 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd508ec6-6de8-432f-8400-9987187af2e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd508ec6-6de8-432f-8400-9987187af2e1" (UID: "dd508ec6-6de8-432f-8400-9987187af2e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:24:07.094638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.094582 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-dshm" (OuterVolumeSpecName: "dshm") pod "dd508ec6-6de8-432f-8400-9987187af2e1" (UID: "dd508ec6-6de8-432f-8400-9987187af2e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:24:07.094720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.094694 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd508ec6-6de8-432f-8400-9987187af2e1-kube-api-access-ph8j8" (OuterVolumeSpecName: "kube-api-access-ph8j8") pod "dd508ec6-6de8-432f-8400-9987187af2e1" (UID: "dd508ec6-6de8-432f-8400-9987187af2e1"). InnerVolumeSpecName "kube-api-access-ph8j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:24:07.192896 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.192863 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd508ec6-6de8-432f-8400-9987187af2e1-dshm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:24:07.192896 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.192892 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph8j8\" (UniqueName: \"kubernetes.io/projected/dd508ec6-6de8-432f-8400-9987187af2e1-kube-api-access-ph8j8\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:24:07.192896 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.192902 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd508ec6-6de8-432f-8400-9987187af2e1-tls-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:24:07.908315 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.908284 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" Sep 07 13:24:07.908754 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.908281 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96" event={"ID":"dd508ec6-6de8-432f-8400-9987187af2e1","Type":"ContainerDied","Data":"24233663e5588576534d688ee3f5a48af00ca5829957286545cb836622d9495d"} Sep 07 13:24:07.908754 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.908419 2565 scope.go:117] "RemoveContainer" containerID="6dce8c03df46dd85b35acac96d990075cae68ecbb7fe84e40fc07ed36ad10706" Sep 07 13:24:07.932511 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.932486 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96"] Sep 07 13:24:07.934419 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:07.934397 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c7c371/sub-test-model-c7c371-kserve-df859-cwj96"] Sep 07 13:24:08.828288 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:08.828250 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd508ec6-6de8-432f-8400-9987187af2e1" path="/var/lib/kubelet/pods/dd508ec6-6de8-432f-8400-9987187af2e1/volumes" Sep 07 13:24:13.179334 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:13.179294 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:13.179840 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:13.179611 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:13.192210 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:13.192189 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:13.950524 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:13.950495 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:24:29.053891 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.053849 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9"] Sep 07 13:24:29.054463 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.054441 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd508ec6-6de8-432f-8400-9987187af2e1" containerName="main" Sep 07 13:24:29.054551 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.054466 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd508ec6-6de8-432f-8400-9987187af2e1" containerName="main" Sep 07 13:24:29.054637 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.054574 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd508ec6-6de8-432f-8400-9987187af2e1" containerName="main" Sep 07 13:24:29.057838 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.057816 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:24:29.060718 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.060698 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-00fbc4\"" Sep 07 13:24:29.070229 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.070207 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9"] Sep 07 13:24:29.199011 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.198974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8xn\" (UniqueName: \"kubernetes.io/projected/cf687a8b-3398-4441-9e51-b58790f2b732-kube-api-access-lt8xn\") pod \"payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9\" (UID: \"cf687a8b-3398-4441-9e51-b58790f2b732\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:24:29.199179 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.199063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf687a8b-3398-4441-9e51-b58790f2b732-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9\" (UID: \"cf687a8b-3398-4441-9e51-b58790f2b732\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:24:29.300084 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.300047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf687a8b-3398-4441-9e51-b58790f2b732-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9\" (UID: \"cf687a8b-3398-4441-9e51-b58790f2b732\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:24:29.300246 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.300140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8xn\" (UniqueName: \"kubernetes.io/projected/cf687a8b-3398-4441-9e51-b58790f2b732-kube-api-access-lt8xn\") pod \"payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9\" (UID: \"cf687a8b-3398-4441-9e51-b58790f2b732\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:24:29.300764 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.300745 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf687a8b-3398-4441-9e51-b58790f2b732-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9\" (UID: \"cf687a8b-3398-4441-9e51-b58790f2b732\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:24:29.309217 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.309159 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8xn\" (UniqueName: \"kubernetes.io/projected/cf687a8b-3398-4441-9e51-b58790f2b732-kube-api-access-lt8xn\") pod \"payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9\" (UID: \"cf687a8b-3398-4441-9e51-b58790f2b732\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:24:29.367805 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.367778 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:24:29.645511 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:29.645439 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9"] Sep 07 13:24:29.648689 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:24:29.648662 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf687a8b_3398_4441_9e51_b58790f2b732.slice/crio-442e3a18e8d749fb034fcb43ad1299967330aa148f6601017438df06b85000a3 WatchSource:0}: Error finding container 442e3a18e8d749fb034fcb43ad1299967330aa148f6601017438df06b85000a3: Status 404 returned error can't find the container with id 442e3a18e8d749fb034fcb43ad1299967330aa148f6601017438df06b85000a3 Sep 07 13:24:30.021279 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:30.021242 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" event={"ID":"cf687a8b-3398-4441-9e51-b58790f2b732","Type":"ContainerStarted","Data":"c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7"} Sep 07 13:24:30.021482 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:30.021294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" event={"ID":"cf687a8b-3398-4441-9e51-b58790f2b732","Type":"ContainerStarted","Data":"442e3a18e8d749fb034fcb43ad1299967330aa148f6601017438df06b85000a3"} Sep 07 13:24:30.021482 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:30.021367 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:24:30.043003 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:30.042950 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" podStartSLOduration=1.042933813 podStartE2EDuration="1.042933813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:24:30.03989882 +0000 UTC m=+1491.720284590" watchObservedRunningTime="2026-09-07 13:24:30.042933813 +0000 UTC m=+1491.723319583" Sep 07 13:24:41.028552 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:41.028517 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:24:57.922228 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:57.922193 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn"] Sep 07 13:24:57.926071 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:57.926048 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:24:57.929277 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:57.929255 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a768a0\"" Sep 07 13:24:57.942941 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:57.942919 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk"] Sep 07 13:24:57.946363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:57.946346 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:24:57.947111 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:57.947090 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn"] Sep 07 13:24:57.948786 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:57.948766 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a768a0-dockercfg-hx52h\"" Sep 07 13:24:57.958360 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:57.958340 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk"] Sep 07 13:24:58.051966 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.051924 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5bdb10-3da7-4d80-9703-8e788eb4800f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn\" (UID: \"cf5bdb10-3da7-4d80-9703-8e788eb4800f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:24:58.051966 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.051967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f0a325-ccef-45f0-a194-afaf5b8cad85-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk\" (UID: \"d1f0a325-ccef-45f0-a194-afaf5b8cad85\") " pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:24:58.052167 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.052053 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws8hj\" (UniqueName: \"kubernetes.io/projected/d1f0a325-ccef-45f0-a194-afaf5b8cad85-kube-api-access-ws8hj\") pod \"payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk\" (UID: \"d1f0a325-ccef-45f0-a194-afaf5b8cad85\") " pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:24:58.052167 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.052103 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9c92\" (UniqueName: \"kubernetes.io/projected/cf5bdb10-3da7-4d80-9703-8e788eb4800f-kube-api-access-x9c92\") pod \"payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn\" (UID: \"cf5bdb10-3da7-4d80-9703-8e788eb4800f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:24:58.152888 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.152858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f0a325-ccef-45f0-a194-afaf5b8cad85-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk\" (UID: \"d1f0a325-ccef-45f0-a194-afaf5b8cad85\") " pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:24:58.153045 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.152940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws8hj\" (UniqueName: \"kubernetes.io/projected/d1f0a325-ccef-45f0-a194-afaf5b8cad85-kube-api-access-ws8hj\") pod \"payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk\" (UID: \"d1f0a325-ccef-45f0-a194-afaf5b8cad85\") " pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:24:58.153045 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.153002 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9c92\" (UniqueName: \"kubernetes.io/projected/cf5bdb10-3da7-4d80-9703-8e788eb4800f-kube-api-access-x9c92\") pod \"payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn\" (UID: \"cf5bdb10-3da7-4d80-9703-8e788eb4800f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:24:58.153132 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.153055 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5bdb10-3da7-4d80-9703-8e788eb4800f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn\" (UID: \"cf5bdb10-3da7-4d80-9703-8e788eb4800f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:24:58.153652 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.153633 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f0a325-ccef-45f0-a194-afaf5b8cad85-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk\" (UID: \"d1f0a325-ccef-45f0-a194-afaf5b8cad85\") " pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:24:58.153747 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.153730 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5bdb10-3da7-4d80-9703-8e788eb4800f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn\" (UID: \"cf5bdb10-3da7-4d80-9703-8e788eb4800f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:24:58.166638 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.166613 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws8hj\" (UniqueName: \"kubernetes.io/projected/d1f0a325-ccef-45f0-a194-afaf5b8cad85-kube-api-access-ws8hj\") pod \"payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk\" (UID: \"d1f0a325-ccef-45f0-a194-afaf5b8cad85\") " pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:24:58.166814 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.166791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9c92\" (UniqueName: \"kubernetes.io/projected/cf5bdb10-3da7-4d80-9703-8e788eb4800f-kube-api-access-x9c92\") pod \"payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn\" (UID: \"cf5bdb10-3da7-4d80-9703-8e788eb4800f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:24:58.235846 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.235775 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:24:58.259015 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.258991 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:24:58.518585 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.518553 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn"] Sep 07 13:24:58.520128 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:24:58.520103 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5bdb10_3da7_4d80_9703_8e788eb4800f.slice/crio-eb938f55cee643dd128ddc03cc6abddc82f3e7a900449cf3f5b67c41a3b41c54 WatchSource:0}: Error finding container eb938f55cee643dd128ddc03cc6abddc82f3e7a900449cf3f5b67c41a3b41c54: Status 404 returned error can't find the container with id eb938f55cee643dd128ddc03cc6abddc82f3e7a900449cf3f5b67c41a3b41c54 Sep 07 13:24:58.552394 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:58.552373 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk"] Sep 07 13:24:58.555735 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:24:58.555714 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f0a325_ccef_45f0_a194_afaf5b8cad85.slice/crio-5cc17d203b00bac3d02920c097b79c7987534fbfc21878720e0393ab2b1cbcbf WatchSource:0}: Error finding container 5cc17d203b00bac3d02920c097b79c7987534fbfc21878720e0393ab2b1cbcbf: Status 404 returned error can't find the container with id 5cc17d203b00bac3d02920c097b79c7987534fbfc21878720e0393ab2b1cbcbf Sep 07 13:24:59.168231 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:59.168188 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" event={"ID":"d1f0a325-ccef-45f0-a194-afaf5b8cad85","Type":"ContainerStarted","Data":"ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32"} Sep 07 13:24:59.168719 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:59.168304 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:24:59.168719 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:59.168334 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" event={"ID":"d1f0a325-ccef-45f0-a194-afaf5b8cad85","Type":"ContainerStarted","Data":"5cc17d203b00bac3d02920c097b79c7987534fbfc21878720e0393ab2b1cbcbf"} Sep 07 13:24:59.169961 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:59.169928 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" event={"ID":"cf5bdb10-3da7-4d80-9703-8e788eb4800f","Type":"ContainerStarted","Data":"f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c"} Sep 07 13:24:59.170067 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:59.169985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" event={"ID":"cf5bdb10-3da7-4d80-9703-8e788eb4800f","Type":"ContainerStarted","Data":"eb938f55cee643dd128ddc03cc6abddc82f3e7a900449cf3f5b67c41a3b41c54"} Sep 07 13:24:59.170067 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:59.170013 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:24:59.190184 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:59.190132 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" podStartSLOduration=2.190113799 podStartE2EDuration="2.190113799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:24:59.18725853 +0000 UTC m=+1520.867644352" watchObservedRunningTime="2026-09-07 13:24:59.190113799 +0000 UTC m=+1520.870499573" Sep 07 13:24:59.209998 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:24:59.209947 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" podStartSLOduration=2.209928246 podStartE2EDuration="2.209928246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:24:59.208186953 +0000 UTC m=+1520.888572724" watchObservedRunningTime="2026-09-07 13:24:59.209928246 +0000 UTC m=+1520.890314020" Sep 07 13:25:10.176958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:10.176872 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:25:10.176958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:10.176956 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:25:23.954407 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:23.954367 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v"] Sep 07 13:25:23.958279 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:23.958258 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v" Sep 07 13:25:23.961842 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:23.961822 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jlvcf\"" Sep 07 13:25:23.963371 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:23.963354 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 13:25:24.002833 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:24.002797 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v"] Sep 07 13:25:24.097223 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:24.097188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea27b57-c86f-4faa-b787-65d8ed8ac274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00fbc4-v259v\" (UID: \"4ea27b57-c86f-4faa-b787-65d8ed8ac274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v" Sep 07 13:25:24.198040 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:24.197996 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea27b57-c86f-4faa-b787-65d8ed8ac274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00fbc4-v259v\" (UID: \"4ea27b57-c86f-4faa-b787-65d8ed8ac274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v" Sep 07 13:25:24.198659 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:24.198636 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea27b57-c86f-4faa-b787-65d8ed8ac274-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-00fbc4-v259v\" (UID: \"4ea27b57-c86f-4faa-b787-65d8ed8ac274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v" Sep 07 13:25:24.271039 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:24.270953 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v" Sep 07 13:25:24.544841 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:24.544812 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v"] Sep 07 13:25:24.547186 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:25:24.547159 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea27b57_c86f_4faa_b787_65d8ed8ac274.slice/crio-0f4bac84299be01823b584d156d25a260d10a2583282b1fcd14fbcce2c85fca9 WatchSource:0}: Error finding container 0f4bac84299be01823b584d156d25a260d10a2583282b1fcd14fbcce2c85fca9: Status 404 returned error can't find the container with id 0f4bac84299be01823b584d156d25a260d10a2583282b1fcd14fbcce2c85fca9 Sep 07 13:25:25.305809 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:25.305782 2565 generic.go:356] "Generic (PLEG): container finished" podID="4ea27b57-c86f-4faa-b787-65d8ed8ac274" containerID="842943a34188c3b4d9b8fed109d4428372462db1820defd0f7732f260ea501ec" exitCode=0 Sep 07 13:25:25.306120 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:25.305878 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v" event={"ID":"4ea27b57-c86f-4faa-b787-65d8ed8ac274","Type":"ContainerDied","Data":"842943a34188c3b4d9b8fed109d4428372462db1820defd0f7732f260ea501ec"} Sep 07 13:25:25.306120 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:25.305927 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v" event={"ID":"4ea27b57-c86f-4faa-b787-65d8ed8ac274","Type":"ContainerStarted","Data":"0f4bac84299be01823b584d156d25a260d10a2583282b1fcd14fbcce2c85fca9"} Sep 07 13:25:26.449794 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:26.449769 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v" Sep 07 13:25:26.621204 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:26.621124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea27b57-c86f-4faa-b787-65d8ed8ac274-maas-api-service-ca\") pod \"4ea27b57-c86f-4faa-b787-65d8ed8ac274\" (UID: \"4ea27b57-c86f-4faa-b787-65d8ed8ac274\") " Sep 07 13:25:26.621485 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:26.621463 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea27b57-c86f-4faa-b787-65d8ed8ac274-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ea27b57-c86f-4faa-b787-65d8ed8ac274" (UID: "4ea27b57-c86f-4faa-b787-65d8ed8ac274"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:25:26.722641 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:26.722521 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea27b57-c86f-4faa-b787-65d8ed8ac274-maas-api-service-ca\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:25:27.317676 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:27.317644 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v" Sep 07 13:25:27.317676 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:27.317656 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v" event={"ID":"4ea27b57-c86f-4faa-b787-65d8ed8ac274","Type":"ContainerDied","Data":"0f4bac84299be01823b584d156d25a260d10a2583282b1fcd14fbcce2c85fca9"} Sep 07 13:25:27.317888 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:27.317713 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f4bac84299be01823b584d156d25a260d10a2583282b1fcd14fbcce2c85fca9" Sep 07 13:25:30.270301 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:30.270266 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p"] Sep 07 13:25:37.724211 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.724172 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5"] Sep 07 13:25:37.724679 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.724663 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea27b57-c86f-4faa-b787-65d8ed8ac274" containerName="revoke-keys" Sep 07 13:25:37.724736 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.724682 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea27b57-c86f-4faa-b787-65d8ed8ac274" containerName="revoke-keys" Sep 07 13:25:37.724773 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.724765 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea27b57-c86f-4faa-b787-65d8ed8ac274" containerName="revoke-keys" Sep 07 13:25:37.728328 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.728307 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:25:37.731633 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.731585 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-219f63-dockercfg-m2klm\"" Sep 07 13:25:37.731760 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.731745 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-219f63\"" Sep 07 13:25:37.751859 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.751830 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5"] Sep 07 13:25:37.836095 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.836058 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzs52\" (UniqueName: \"kubernetes.io/projected/d55f934f-38bb-4ea8-b721-874ab9845cdb-kube-api-access-bzs52\") pod \"payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5\" (UID: \"d55f934f-38bb-4ea8-b721-874ab9845cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:25:37.836095 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.836096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55f934f-38bb-4ea8-b721-874ab9845cdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5\" (UID: \"d55f934f-38bb-4ea8-b721-874ab9845cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:25:37.936742 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.936707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzs52\" (UniqueName: \"kubernetes.io/projected/d55f934f-38bb-4ea8-b721-874ab9845cdb-kube-api-access-bzs52\") pod \"payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5\" (UID: \"d55f934f-38bb-4ea8-b721-874ab9845cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:25:37.936742 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.936743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55f934f-38bb-4ea8-b721-874ab9845cdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5\" (UID: \"d55f934f-38bb-4ea8-b721-874ab9845cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:25:37.937436 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.937414 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55f934f-38bb-4ea8-b721-874ab9845cdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5\" (UID: \"d55f934f-38bb-4ea8-b721-874ab9845cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:25:37.952865 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:37.952832 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzs52\" (UniqueName: \"kubernetes.io/projected/d55f934f-38bb-4ea8-b721-874ab9845cdb-kube-api-access-bzs52\") pod \"payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5\" (UID: \"d55f934f-38bb-4ea8-b721-874ab9845cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:25:38.039884 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:38.039796 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:25:38.325440 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:38.325414 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5"] Sep 07 13:25:38.327131 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:25:38.327104 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55f934f_38bb_4ea8_b721_874ab9845cdb.slice/crio-980fa11b5ea1bdeed9b6ce23516a2a768f22c6962a87fe0a5bfb1f2381f4039d WatchSource:0}: Error finding container 980fa11b5ea1bdeed9b6ce23516a2a768f22c6962a87fe0a5bfb1f2381f4039d: Status 404 returned error can't find the container with id 980fa11b5ea1bdeed9b6ce23516a2a768f22c6962a87fe0a5bfb1f2381f4039d Sep 07 13:25:38.374375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:38.374347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" event={"ID":"d55f934f-38bb-4ea8-b721-874ab9845cdb","Type":"ContainerStarted","Data":"980fa11b5ea1bdeed9b6ce23516a2a768f22c6962a87fe0a5bfb1f2381f4039d"} Sep 07 13:25:39.380824 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:39.380793 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" event={"ID":"d55f934f-38bb-4ea8-b721-874ab9845cdb","Type":"ContainerStarted","Data":"9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703"} Sep 07 13:25:39.381282 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:39.380862 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:25:39.407150 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:39.407101 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" podStartSLOduration=2.407086593 podStartE2EDuration="2.407086593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:25:39.405404188 +0000 UTC m=+1561.085789959" watchObservedRunningTime="2026-09-07 13:25:39.407086593 +0000 UTC m=+1561.087472420" Sep 07 13:25:40.358014 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.357980 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9"] Sep 07 13:25:40.358249 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.358228 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" podUID="cf687a8b-3398-4441-9e51-b58790f2b732" containerName="payload-pre-processing" containerID="cri-o://c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7" gracePeriod=30 Sep 07 13:25:40.597063 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.597031 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v"] Sep 07 13:25:40.601160 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.601136 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-00fbc4-v259v"] Sep 07 13:25:40.616408 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.616358 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:25:40.665238 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.665210 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf687a8b-3398-4441-9e51-b58790f2b732-plugins-config-volume\") pod \"cf687a8b-3398-4441-9e51-b58790f2b732\" (UID: \"cf687a8b-3398-4441-9e51-b58790f2b732\") " Sep 07 13:25:40.665418 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.665250 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt8xn\" (UniqueName: \"kubernetes.io/projected/cf687a8b-3398-4441-9e51-b58790f2b732-kube-api-access-lt8xn\") pod \"cf687a8b-3398-4441-9e51-b58790f2b732\" (UID: \"cf687a8b-3398-4441-9e51-b58790f2b732\") " Sep 07 13:25:40.665586 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.665562 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf687a8b-3398-4441-9e51-b58790f2b732-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf687a8b-3398-4441-9e51-b58790f2b732" (UID: "cf687a8b-3398-4441-9e51-b58790f2b732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:25:40.665704 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.665688 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf687a8b-3398-4441-9e51-b58790f2b732-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:25:40.667450 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.667426 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf687a8b-3398-4441-9e51-b58790f2b732-kube-api-access-lt8xn" (OuterVolumeSpecName: "kube-api-access-lt8xn") pod "cf687a8b-3398-4441-9e51-b58790f2b732" (UID: "cf687a8b-3398-4441-9e51-b58790f2b732"). InnerVolumeSpecName "kube-api-access-lt8xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:25:40.767023 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.766983 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8xn\" (UniqueName: \"kubernetes.io/projected/cf687a8b-3398-4441-9e51-b58790f2b732-kube-api-access-lt8xn\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:25:40.827513 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:40.827478 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea27b57-c86f-4faa-b787-65d8ed8ac274" path="/var/lib/kubelet/pods/4ea27b57-c86f-4faa-b787-65d8ed8ac274/volumes" Sep 07 13:25:41.393351 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:41.393318 2565 generic.go:356] "Generic (PLEG): container finished" podID="cf687a8b-3398-4441-9e51-b58790f2b732" containerID="c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7" exitCode=0 Sep 07 13:25:41.393532 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:41.393358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" event={"ID":"cf687a8b-3398-4441-9e51-b58790f2b732","Type":"ContainerDied","Data":"c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7"} Sep 07 13:25:41.393532 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:41.393365 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" Sep 07 13:25:41.393532 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:41.393402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9" event={"ID":"cf687a8b-3398-4441-9e51-b58790f2b732","Type":"ContainerDied","Data":"442e3a18e8d749fb034fcb43ad1299967330aa148f6601017438df06b85000a3"} Sep 07 13:25:41.393532 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:41.393430 2565 scope.go:117] "RemoveContainer" containerID="c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7" Sep 07 13:25:41.402888 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:41.402868 2565 scope.go:117] "RemoveContainer" containerID="c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7" Sep 07 13:25:41.403154 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:25:41.403133 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7\": container with ID starting with c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7 not found: ID does not exist" containerID="c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7" Sep 07 13:25:41.403197 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:41.403167 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7"} err="failed to get container status \"c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7\": rpc error: code = NotFound desc = could not find container \"c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7\": container with ID starting with c1a99a4c05d07b5163359977d39eccada0523420d0cd28410bfabdc190453eb7 not found: ID does not exist" Sep 07 13:25:41.415482 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:41.415452 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9"] Sep 07 13:25:41.420352 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:41.420330 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-00fbc4-59b99c9686-cvdm9"] Sep 07 13:25:42.827660 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:42.827630 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf687a8b-3398-4441-9e51-b58790f2b732" path="/var/lib/kubelet/pods/cf687a8b-3398-4441-9e51-b58790f2b732/volumes" Sep 07 13:25:45.289647 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.289573 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" podUID="542040c9-b85c-43b5-b6b2-c2daf66f3ac6" containerName="main" containerID="cri-o://7722cdd00eff64b9bf4e72dc9e310652db117394802d132f3492c5bb3aff0f63" gracePeriod=45 Sep 07 13:25:45.417705 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.417675 2565 generic.go:356] "Generic (PLEG): container finished" podID="542040c9-b85c-43b5-b6b2-c2daf66f3ac6" containerID="7722cdd00eff64b9bf4e72dc9e310652db117394802d132f3492c5bb3aff0f63" exitCode=0 Sep 07 13:25:45.417883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.417733 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" event={"ID":"542040c9-b85c-43b5-b6b2-c2daf66f3ac6","Type":"ContainerDied","Data":"7722cdd00eff64b9bf4e72dc9e310652db117394802d132f3492c5bb3aff0f63"} Sep 07 13:25:45.547126 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.547068 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:25:45.614128 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614093 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b59z\" (UniqueName: \"kubernetes.io/projected/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-kube-api-access-5b59z\") pod \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " Sep 07 13:25:45.614274 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614199 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tmp-dir\") pod \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " Sep 07 13:25:45.614274 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614227 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-home\") pod \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " Sep 07 13:25:45.614274 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614252 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-dshm\") pod \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " Sep 07 13:25:45.614403 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614302 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-model-cache\") pod \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " Sep 07 13:25:45.614403 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614330 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tls-certs\") pod \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\" (UID: \"542040c9-b85c-43b5-b6b2-c2daf66f3ac6\") " Sep 07 13:25:45.614515 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614451 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-home" (OuterVolumeSpecName: "home") pod "542040c9-b85c-43b5-b6b2-c2daf66f3ac6" (UID: "542040c9-b85c-43b5-b6b2-c2daf66f3ac6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:25:45.614515 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614500 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "542040c9-b85c-43b5-b6b2-c2daf66f3ac6" (UID: "542040c9-b85c-43b5-b6b2-c2daf66f3ac6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:25:45.614663 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614584 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-model-cache" (OuterVolumeSpecName: "model-cache") pod "542040c9-b85c-43b5-b6b2-c2daf66f3ac6" (UID: "542040c9-b85c-43b5-b6b2-c2daf66f3ac6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:25:45.614788 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614765 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tmp-dir\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:25:45.614827 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614794 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-home\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:25:45.614827 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.614807 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-model-cache\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:25:45.616331 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.616309 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-dshm" (OuterVolumeSpecName: "dshm") pod "542040c9-b85c-43b5-b6b2-c2daf66f3ac6" (UID: "542040c9-b85c-43b5-b6b2-c2daf66f3ac6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:25:45.616426 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.616397 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-kube-api-access-5b59z" (OuterVolumeSpecName: "kube-api-access-5b59z") pod "542040c9-b85c-43b5-b6b2-c2daf66f3ac6" (UID: "542040c9-b85c-43b5-b6b2-c2daf66f3ac6"). InnerVolumeSpecName "kube-api-access-5b59z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:25:45.616504 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.616479 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "542040c9-b85c-43b5-b6b2-c2daf66f3ac6" (UID: "542040c9-b85c-43b5-b6b2-c2daf66f3ac6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:25:45.715778 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.715743 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-dshm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:25:45.715778 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.715772 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-tls-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:25:45.715778 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:45.715782 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b59z\" (UniqueName: \"kubernetes.io/projected/542040c9-b85c-43b5-b6b2-c2daf66f3ac6-kube-api-access-5b59z\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:25:46.425402 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:46.425369 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" Sep 07 13:25:46.425402 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:46.425392 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p" event={"ID":"542040c9-b85c-43b5-b6b2-c2daf66f3ac6","Type":"ContainerDied","Data":"329d7217149c7d14eea7cc6378c5049f16060c177b502a7976f1f2d7e5bc6e2a"} Sep 07 13:25:46.425975 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:46.425441 2565 scope.go:117] "RemoveContainer" containerID="7722cdd00eff64b9bf4e72dc9e310652db117394802d132f3492c5bb3aff0f63" Sep 07 13:25:46.452049 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:46.452018 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p"] Sep 07 13:25:46.455587 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:46.455562 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5910bb/rate-test-model-5910bb-kserve-ff5dcb9c9-qwd5p"] Sep 07 13:25:46.828554 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:46.828475 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="542040c9-b85c-43b5-b6b2-c2daf66f3ac6" path="/var/lib/kubelet/pods/542040c9-b85c-43b5-b6b2-c2daf66f3ac6/volumes" Sep 07 13:25:48.550104 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.550071 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8"] Sep 07 13:25:48.550498 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.550486 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542040c9-b85c-43b5-b6b2-c2daf66f3ac6" containerName="main" Sep 07 13:25:48.550555 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.550500 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="542040c9-b85c-43b5-b6b2-c2daf66f3ac6" containerName="main" Sep 07 13:25:48.550555 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.550519 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf687a8b-3398-4441-9e51-b58790f2b732" containerName="payload-pre-processing" Sep 07 13:25:48.550555 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.550524 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf687a8b-3398-4441-9e51-b58790f2b732" containerName="payload-pre-processing" Sep 07 13:25:48.550687 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.550608 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="542040c9-b85c-43b5-b6b2-c2daf66f3ac6" containerName="main" Sep 07 13:25:48.550687 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.550616 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf687a8b-3398-4441-9e51-b58790f2b732" containerName="payload-pre-processing" Sep 07 13:25:48.553783 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.553767 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" Sep 07 13:25:48.557127 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.557107 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 13:25:48.557340 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.557323 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jlvcf\"" Sep 07 13:25:48.575480 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.575457 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8"] Sep 07 13:25:48.646367 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.646332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9d6a27-979b-4096-855b-16b4bfb18124-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a768a0-56lp8\" (UID: \"2d9d6a27-979b-4096-855b-16b4bfb18124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" Sep 07 13:25:48.747243 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.747214 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9d6a27-979b-4096-855b-16b4bfb18124-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a768a0-56lp8\" (UID: \"2d9d6a27-979b-4096-855b-16b4bfb18124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" Sep 07 13:25:48.747863 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.747842 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9d6a27-979b-4096-855b-16b4bfb18124-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a768a0-56lp8\" (UID: \"2d9d6a27-979b-4096-855b-16b4bfb18124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" Sep 07 13:25:48.864614 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:48.864515 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" Sep 07 13:25:49.141560 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:49.141531 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8"] Sep 07 13:25:49.142410 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:25:49.142385 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9d6a27_979b_4096_855b_16b4bfb18124.slice/crio-67d34aca04cfde9145982fd954bb2695b52533b243182d9b663904e46aebd6dd WatchSource:0}: Error finding container 67d34aca04cfde9145982fd954bb2695b52533b243182d9b663904e46aebd6dd: Status 404 returned error can't find the container with id 67d34aca04cfde9145982fd954bb2695b52533b243182d9b663904e46aebd6dd Sep 07 13:25:49.443700 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:49.443668 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" event={"ID":"2d9d6a27-979b-4096-855b-16b4bfb18124","Type":"ContainerStarted","Data":"0b71826d4fa8c2fe1b86c36f726e2dd27c579b5006a27ab465c7873555f4e16e"} Sep 07 13:25:49.443865 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:49.443725 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" event={"ID":"2d9d6a27-979b-4096-855b-16b4bfb18124","Type":"ContainerStarted","Data":"67d34aca04cfde9145982fd954bb2695b52533b243182d9b663904e46aebd6dd"} Sep 07 13:25:49.469963 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:49.469907 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" podStartSLOduration=1.469892164 podStartE2EDuration="1.469892164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:25:49.46830631 +0000 UTC m=+1571.148692082" watchObservedRunningTime="2026-09-07 13:25:49.469892164 +0000 UTC m=+1571.150277971" Sep 07 13:25:50.389214 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:50.389175 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:25:50.452521 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:50.452490 2565 generic.go:356] "Generic (PLEG): container finished" podID="2d9d6a27-979b-4096-855b-16b4bfb18124" containerID="0b71826d4fa8c2fe1b86c36f726e2dd27c579b5006a27ab465c7873555f4e16e" exitCode=0 Sep 07 13:25:50.452715 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:50.452572 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" event={"ID":"2d9d6a27-979b-4096-855b-16b4bfb18124","Type":"ContainerDied","Data":"0b71826d4fa8c2fe1b86c36f726e2dd27c579b5006a27ab465c7873555f4e16e"} Sep 07 13:25:51.597902 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:51.597877 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" Sep 07 13:25:51.673192 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:51.673158 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9d6a27-979b-4096-855b-16b4bfb18124-maas-api-service-ca\") pod \"2d9d6a27-979b-4096-855b-16b4bfb18124\" (UID: \"2d9d6a27-979b-4096-855b-16b4bfb18124\") " Sep 07 13:25:51.673512 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:51.673489 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9d6a27-979b-4096-855b-16b4bfb18124-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d9d6a27-979b-4096-855b-16b4bfb18124" (UID: "2d9d6a27-979b-4096-855b-16b4bfb18124"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:25:51.774090 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:51.773996 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d9d6a27-979b-4096-855b-16b4bfb18124-maas-api-service-ca\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:25:52.464907 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:52.464875 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" Sep 07 13:25:52.464907 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:52.464888 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8" event={"ID":"2d9d6a27-979b-4096-855b-16b4bfb18124","Type":"ContainerDied","Data":"67d34aca04cfde9145982fd954bb2695b52533b243182d9b663904e46aebd6dd"} Sep 07 13:25:52.465116 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:25:52.464928 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d34aca04cfde9145982fd954bb2695b52533b243182d9b663904e46aebd6dd" Sep 07 13:26:03.927361 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:03.927321 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk"] Sep 07 13:26:03.927913 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:03.927520 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" podUID="d1f0a325-ccef-45f0-a194-afaf5b8cad85" containerName="payload-processing" containerID="cri-o://ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32" gracePeriod=30 Sep 07 13:26:03.944480 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:03.944450 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn"] Sep 07 13:26:03.944730 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:03.944706 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" podUID="cf5bdb10-3da7-4d80-9703-8e788eb4800f" containerName="payload-pre-processing" containerID="cri-o://f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c" gracePeriod=30 Sep 07 13:26:04.198547 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.198519 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8"] Sep 07 13:26:04.204329 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.204307 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a768a0-56lp8"] Sep 07 13:26:04.210549 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.210527 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:26:04.226675 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.226652 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:26:04.394530 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.394490 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5bdb10-3da7-4d80-9703-8e788eb4800f-plugins-config-volume\") pod \"cf5bdb10-3da7-4d80-9703-8e788eb4800f\" (UID: \"cf5bdb10-3da7-4d80-9703-8e788eb4800f\") " Sep 07 13:26:04.394720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.394649 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws8hj\" (UniqueName: \"kubernetes.io/projected/d1f0a325-ccef-45f0-a194-afaf5b8cad85-kube-api-access-ws8hj\") pod \"d1f0a325-ccef-45f0-a194-afaf5b8cad85\" (UID: \"d1f0a325-ccef-45f0-a194-afaf5b8cad85\") " Sep 07 13:26:04.394720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.394693 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9c92\" (UniqueName: \"kubernetes.io/projected/cf5bdb10-3da7-4d80-9703-8e788eb4800f-kube-api-access-x9c92\") pod \"cf5bdb10-3da7-4d80-9703-8e788eb4800f\" (UID: \"cf5bdb10-3da7-4d80-9703-8e788eb4800f\") " Sep 07 13:26:04.394842 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.394736 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f0a325-ccef-45f0-a194-afaf5b8cad85-plugins-config-volume\") pod \"d1f0a325-ccef-45f0-a194-afaf5b8cad85\" (UID: \"d1f0a325-ccef-45f0-a194-afaf5b8cad85\") " Sep 07 13:26:04.394984 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.394948 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5bdb10-3da7-4d80-9703-8e788eb4800f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf5bdb10-3da7-4d80-9703-8e788eb4800f" (UID: "cf5bdb10-3da7-4d80-9703-8e788eb4800f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:26:04.395159 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.395143 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5bdb10-3da7-4d80-9703-8e788eb4800f-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:04.395268 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.395244 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f0a325-ccef-45f0-a194-afaf5b8cad85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1f0a325-ccef-45f0-a194-afaf5b8cad85" (UID: "d1f0a325-ccef-45f0-a194-afaf5b8cad85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:26:04.396806 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.396785 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f0a325-ccef-45f0-a194-afaf5b8cad85-kube-api-access-ws8hj" (OuterVolumeSpecName: "kube-api-access-ws8hj") pod "d1f0a325-ccef-45f0-a194-afaf5b8cad85" (UID: "d1f0a325-ccef-45f0-a194-afaf5b8cad85"). InnerVolumeSpecName "kube-api-access-ws8hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:26:04.396869 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.396803 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5bdb10-3da7-4d80-9703-8e788eb4800f-kube-api-access-x9c92" (OuterVolumeSpecName: "kube-api-access-x9c92") pod "cf5bdb10-3da7-4d80-9703-8e788eb4800f" (UID: "cf5bdb10-3da7-4d80-9703-8e788eb4800f"). InnerVolumeSpecName "kube-api-access-x9c92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:26:04.496709 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.496631 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9c92\" (UniqueName: \"kubernetes.io/projected/cf5bdb10-3da7-4d80-9703-8e788eb4800f-kube-api-access-x9c92\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:04.496709 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.496655 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f0a325-ccef-45f0-a194-afaf5b8cad85-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:04.496709 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.496665 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws8hj\" (UniqueName: \"kubernetes.io/projected/d1f0a325-ccef-45f0-a194-afaf5b8cad85-kube-api-access-ws8hj\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:04.529344 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.529308 2565 generic.go:356] "Generic (PLEG): container finished" podID="d1f0a325-ccef-45f0-a194-afaf5b8cad85" containerID="ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32" exitCode=0 Sep 07 13:26:04.529502 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.529356 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" Sep 07 13:26:04.529502 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.529386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" event={"ID":"d1f0a325-ccef-45f0-a194-afaf5b8cad85","Type":"ContainerDied","Data":"ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32"} Sep 07 13:26:04.529502 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.529444 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk" event={"ID":"d1f0a325-ccef-45f0-a194-afaf5b8cad85","Type":"ContainerDied","Data":"5cc17d203b00bac3d02920c097b79c7987534fbfc21878720e0393ab2b1cbcbf"} Sep 07 13:26:04.529502 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.529479 2565 scope.go:117] "RemoveContainer" containerID="ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32" Sep 07 13:26:04.530709 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.530688 2565 generic.go:356] "Generic (PLEG): container finished" podID="cf5bdb10-3da7-4d80-9703-8e788eb4800f" containerID="f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c" exitCode=0 Sep 07 13:26:04.530818 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.530728 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" Sep 07 13:26:04.530879 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.530731 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" event={"ID":"cf5bdb10-3da7-4d80-9703-8e788eb4800f","Type":"ContainerDied","Data":"f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c"} Sep 07 13:26:04.530879 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.530855 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn" event={"ID":"cf5bdb10-3da7-4d80-9703-8e788eb4800f","Type":"ContainerDied","Data":"eb938f55cee643dd128ddc03cc6abddc82f3e7a900449cf3f5b67c41a3b41c54"} Sep 07 13:26:04.539151 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.539135 2565 scope.go:117] "RemoveContainer" containerID="ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32" Sep 07 13:26:04.539435 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:26:04.539416 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32\": container with ID starting with ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32 not found: ID does not exist" containerID="ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32" Sep 07 13:26:04.539512 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.539446 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32"} err="failed to get container status \"ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32\": rpc error: code = NotFound desc = could not find container \"ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32\": container with ID starting with ffe3f25496685a7e5502928a86ef0009e0e7508e3f3e4f5159236124ae693d32 not found: ID does not exist" Sep 07 13:26:04.539512 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.539469 2565 scope.go:117] "RemoveContainer" containerID="f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c" Sep 07 13:26:04.547367 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.547350 2565 scope.go:117] "RemoveContainer" containerID="f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c" Sep 07 13:26:04.547648 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:26:04.547625 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c\": container with ID starting with f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c not found: ID does not exist" containerID="f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c" Sep 07 13:26:04.547708 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.547647 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c"} err="failed to get container status \"f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c\": rpc error: code = NotFound desc = could not find container \"f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c\": container with ID starting with f9d9321a30469f45c88a31866886b4b0334378daec89c59a6fd1ceb4b211e80c not found: ID does not exist" Sep 07 13:26:04.553640 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.553619 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk"] Sep 07 13:26:04.561043 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.561023 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a768a0-7d8554c69f-cljrk"] Sep 07 13:26:04.574614 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.574581 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn"] Sep 07 13:26:04.578016 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.577998 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a768a0-956b496bb-mmwsn"] Sep 07 13:26:04.826853 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.826759 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9d6a27-979b-4096-855b-16b4bfb18124" path="/var/lib/kubelet/pods/2d9d6a27-979b-4096-855b-16b4bfb18124/volumes" Sep 07 13:26:04.827233 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.827212 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf5bdb10-3da7-4d80-9703-8e788eb4800f" path="/var/lib/kubelet/pods/cf5bdb10-3da7-4d80-9703-8e788eb4800f/volumes" Sep 07 13:26:04.827626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:04.827591 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1f0a325-ccef-45f0-a194-afaf5b8cad85" path="/var/lib/kubelet/pods/d1f0a325-ccef-45f0-a194-afaf5b8cad85/volumes" Sep 07 13:26:07.710446 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.710403 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq"] Sep 07 13:26:07.711833 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.711803 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5bdb10-3da7-4d80-9703-8e788eb4800f" containerName="payload-pre-processing" Sep 07 13:26:07.712006 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.711993 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5bdb10-3da7-4d80-9703-8e788eb4800f" containerName="payload-pre-processing" Sep 07 13:26:07.712124 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.712114 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9d6a27-979b-4096-855b-16b4bfb18124" containerName="revoke-keys" Sep 07 13:26:07.712204 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.712195 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9d6a27-979b-4096-855b-16b4bfb18124" containerName="revoke-keys" Sep 07 13:26:07.712292 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.712282 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f0a325-ccef-45f0-a194-afaf5b8cad85" containerName="payload-processing" Sep 07 13:26:07.712372 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.712363 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f0a325-ccef-45f0-a194-afaf5b8cad85" containerName="payload-processing" Sep 07 13:26:07.712662 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.712647 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5bdb10-3da7-4d80-9703-8e788eb4800f" containerName="payload-pre-processing" Sep 07 13:26:07.712772 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.712762 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f0a325-ccef-45f0-a194-afaf5b8cad85" containerName="payload-processing" Sep 07 13:26:07.712845 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.712834 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9d6a27-979b-4096-855b-16b4bfb18124" containerName="revoke-keys" Sep 07 13:26:07.718404 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.718381 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:26:07.722304 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.722283 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8dc72a\"" Sep 07 13:26:07.723811 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.723786 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq"] Sep 07 13:26:07.726978 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.726948 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq\" (UID: \"334fe76b-ad10-4ff5-b64a-da7ba597b1b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:26:07.727091 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.727009 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-924xn\" (UniqueName: \"kubernetes.io/projected/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-kube-api-access-924xn\") pod \"payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq\" (UID: \"334fe76b-ad10-4ff5-b64a-da7ba597b1b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:26:07.827681 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.827646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq\" (UID: \"334fe76b-ad10-4ff5-b64a-da7ba597b1b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:26:07.827844 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.827693 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-924xn\" (UniqueName: \"kubernetes.io/projected/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-kube-api-access-924xn\") pod \"payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq\" (UID: \"334fe76b-ad10-4ff5-b64a-da7ba597b1b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:26:07.828363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.828344 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq\" (UID: \"334fe76b-ad10-4ff5-b64a-da7ba597b1b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:26:07.836046 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:07.836023 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-924xn\" (UniqueName: \"kubernetes.io/projected/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-kube-api-access-924xn\") pod \"payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq\" (UID: \"334fe76b-ad10-4ff5-b64a-da7ba597b1b4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:26:08.029754 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:08.029651 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:26:08.314132 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:08.314056 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq"] Sep 07 13:26:08.316617 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:26:08.316565 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334fe76b_ad10_4ff5_b64a_da7ba597b1b4.slice/crio-7bb0440f1d3a72b733fb6f93142f6c05f03e9d7345cb451498c2757077d6decf WatchSource:0}: Error finding container 7bb0440f1d3a72b733fb6f93142f6c05f03e9d7345cb451498c2757077d6decf: Status 404 returned error can't find the container with id 7bb0440f1d3a72b733fb6f93142f6c05f03e9d7345cb451498c2757077d6decf Sep 07 13:26:08.556928 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:08.556886 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" event={"ID":"334fe76b-ad10-4ff5-b64a-da7ba597b1b4","Type":"ContainerStarted","Data":"7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5"} Sep 07 13:26:08.557078 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:08.556943 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" event={"ID":"334fe76b-ad10-4ff5-b64a-da7ba597b1b4","Type":"ContainerStarted","Data":"7bb0440f1d3a72b733fb6f93142f6c05f03e9d7345cb451498c2757077d6decf"} Sep 07 13:26:08.557078 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:08.556974 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:26:11.620984 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:11.620928 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" podStartSLOduration=4.620909915 podStartE2EDuration="4.620909915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:26:08.612726529 +0000 UTC m=+1590.293112300" watchObservedRunningTime="2026-09-07 13:26:11.620909915 +0000 UTC m=+1593.301295716" Sep 07 13:26:11.621413 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:11.621395 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz"] Sep 07 13:26:11.625252 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:11.625233 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" Sep 07 13:26:11.627733 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:11.627709 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 13:26:11.627844 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:11.627709 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jlvcf\"" Sep 07 13:26:11.635218 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:11.635194 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz"] Sep 07 13:26:11.663855 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:11.663821 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0df32e-8334-442f-9460-46bb6b2e4923-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz\" (UID: \"be0df32e-8334-442f-9460-46bb6b2e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" Sep 07 13:26:11.764633 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:11.764576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0df32e-8334-442f-9460-46bb6b2e4923-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz\" (UID: \"be0df32e-8334-442f-9460-46bb6b2e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" Sep 07 13:26:11.765205 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:11.765187 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0df32e-8334-442f-9460-46bb6b2e4923-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz\" (UID: \"be0df32e-8334-442f-9460-46bb6b2e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" Sep 07 13:26:11.936472 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:11.936441 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" Sep 07 13:26:12.218891 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:12.218865 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz"] Sep 07 13:26:12.220470 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:26:12.220446 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0df32e_8334_442f_9460_46bb6b2e4923.slice/crio-87e4a2fabd63db9797fd507dd382c871589cc26bc1a367471f7b3a9b38293462 WatchSource:0}: Error finding container 87e4a2fabd63db9797fd507dd382c871589cc26bc1a367471f7b3a9b38293462: Status 404 returned error can't find the container with id 87e4a2fabd63db9797fd507dd382c871589cc26bc1a367471f7b3a9b38293462 Sep 07 13:26:12.578793 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:12.578756 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" event={"ID":"be0df32e-8334-442f-9460-46bb6b2e4923","Type":"ContainerStarted","Data":"5595f6b08108951735d2c8ef4d2e1e60736896535ca2adb611442d7976342eb4"} Sep 07 13:26:12.578936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:12.578810 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" event={"ID":"be0df32e-8334-442f-9460-46bb6b2e4923","Type":"ContainerStarted","Data":"87e4a2fabd63db9797fd507dd382c871589cc26bc1a367471f7b3a9b38293462"} Sep 07 13:26:12.603994 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:12.603937 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" podStartSLOduration=1.6039219409999999 podStartE2EDuration="1.603921941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:26:12.600820271 +0000 UTC m=+1594.281206045" watchObservedRunningTime="2026-09-07 13:26:12.603921941 +0000 UTC m=+1594.284307710" Sep 07 13:26:13.585561 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:13.585525 2565 generic.go:356] "Generic (PLEG): container finished" podID="be0df32e-8334-442f-9460-46bb6b2e4923" containerID="5595f6b08108951735d2c8ef4d2e1e60736896535ca2adb611442d7976342eb4" exitCode=0 Sep 07 13:26:13.585990 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:13.585622 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" event={"ID":"be0df32e-8334-442f-9460-46bb6b2e4923","Type":"ContainerDied","Data":"5595f6b08108951735d2c8ef4d2e1e60736896535ca2adb611442d7976342eb4"} Sep 07 13:26:14.727482 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:14.727452 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" Sep 07 13:26:14.791908 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:14.791874 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0df32e-8334-442f-9460-46bb6b2e4923-maas-api-service-ca\") pod \"be0df32e-8334-442f-9460-46bb6b2e4923\" (UID: \"be0df32e-8334-442f-9460-46bb6b2e4923\") " Sep 07 13:26:14.792258 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:14.792238 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be0df32e-8334-442f-9460-46bb6b2e4923-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be0df32e-8334-442f-9460-46bb6b2e4923" (UID: "be0df32e-8334-442f-9460-46bb6b2e4923"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:26:14.893658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:14.893566 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be0df32e-8334-442f-9460-46bb6b2e4923-maas-api-service-ca\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:15.598398 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:15.598360 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" Sep 07 13:26:15.598398 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:15.598382 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz" event={"ID":"be0df32e-8334-442f-9460-46bb6b2e4923","Type":"ContainerDied","Data":"87e4a2fabd63db9797fd507dd382c871589cc26bc1a367471f7b3a9b38293462"} Sep 07 13:26:15.598630 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:15.598433 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e4a2fabd63db9797fd507dd382c871589cc26bc1a367471f7b3a9b38293462" Sep 07 13:26:19.563837 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:19.563805 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:26:22.405462 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.405425 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6"] Sep 07 13:26:22.406054 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.405709 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" podUID="5042b6d7-912b-4ef1-ace2-5446a290c71d" containerName="payload-processing" containerID="cri-o://ccae08fecf1007b6b5a9cbf1d23c8def84c1a32b13ee9861302aa47a676dda1c" gracePeriod=30 Sep 07 13:26:22.642515 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.642489 2565 generic.go:356] "Generic (PLEG): container finished" podID="5042b6d7-912b-4ef1-ace2-5446a290c71d" containerID="ccae08fecf1007b6b5a9cbf1d23c8def84c1a32b13ee9861302aa47a676dda1c" exitCode=0 Sep 07 13:26:22.642685 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.642540 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" event={"ID":"5042b6d7-912b-4ef1-ace2-5446a290c71d","Type":"ContainerDied","Data":"ccae08fecf1007b6b5a9cbf1d23c8def84c1a32b13ee9861302aa47a676dda1c"} Sep 07 13:26:22.644104 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.644081 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz"] Sep 07 13:26:22.646876 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.646851 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d5da3f-qwfbz"] Sep 07 13:26:22.661181 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.661110 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:26:22.771979 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.771940 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcgrq\" (UniqueName: \"kubernetes.io/projected/5042b6d7-912b-4ef1-ace2-5446a290c71d-kube-api-access-zcgrq\") pod \"5042b6d7-912b-4ef1-ace2-5446a290c71d\" (UID: \"5042b6d7-912b-4ef1-ace2-5446a290c71d\") " Sep 07 13:26:22.771979 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.771983 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5042b6d7-912b-4ef1-ace2-5446a290c71d-plugins-config-volume\") pod \"5042b6d7-912b-4ef1-ace2-5446a290c71d\" (UID: \"5042b6d7-912b-4ef1-ace2-5446a290c71d\") " Sep 07 13:26:22.772418 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.772392 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5042b6d7-912b-4ef1-ace2-5446a290c71d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5042b6d7-912b-4ef1-ace2-5446a290c71d" (UID: "5042b6d7-912b-4ef1-ace2-5446a290c71d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:26:22.774006 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.773981 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5042b6d7-912b-4ef1-ace2-5446a290c71d-kube-api-access-zcgrq" (OuterVolumeSpecName: "kube-api-access-zcgrq") pod "5042b6d7-912b-4ef1-ace2-5446a290c71d" (UID: "5042b6d7-912b-4ef1-ace2-5446a290c71d"). InnerVolumeSpecName "kube-api-access-zcgrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:26:22.827809 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.827777 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be0df32e-8334-442f-9460-46bb6b2e4923" path="/var/lib/kubelet/pods/be0df32e-8334-442f-9460-46bb6b2e4923/volumes" Sep 07 13:26:22.873515 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.873473 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcgrq\" (UniqueName: \"kubernetes.io/projected/5042b6d7-912b-4ef1-ace2-5446a290c71d-kube-api-access-zcgrq\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:22.873515 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:22.873505 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5042b6d7-912b-4ef1-ace2-5446a290c71d-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:23.650497 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:23.650467 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" Sep 07 13:26:23.650943 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:23.650471 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6" event={"ID":"5042b6d7-912b-4ef1-ace2-5446a290c71d","Type":"ContainerDied","Data":"93bf02f6d22cda378b15582d5e8e1be515fe7ecb5770a1f41bbd6349d5554c36"} Sep 07 13:26:23.650943 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:23.650655 2565 scope.go:117] "RemoveContainer" containerID="ccae08fecf1007b6b5a9cbf1d23c8def84c1a32b13ee9861302aa47a676dda1c" Sep 07 13:26:23.669611 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:23.669568 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6"] Sep 07 13:26:23.674816 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:23.674790 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d5da3f-5668f4d967-6mmx6"] Sep 07 13:26:24.826752 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:24.826718 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5042b6d7-912b-4ef1-ace2-5446a290c71d" path="/var/lib/kubelet/pods/5042b6d7-912b-4ef1-ace2-5446a290c71d/volumes" Sep 07 13:26:31.044974 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.044933 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp"] Sep 07 13:26:31.045381 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.045371 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be0df32e-8334-442f-9460-46bb6b2e4923" containerName="revoke-keys" Sep 07 13:26:31.045431 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.045383 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="be0df32e-8334-442f-9460-46bb6b2e4923" containerName="revoke-keys" Sep 07 13:26:31.045431 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.045396 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5042b6d7-912b-4ef1-ace2-5446a290c71d" containerName="payload-processing" Sep 07 13:26:31.045431 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.045402 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5042b6d7-912b-4ef1-ace2-5446a290c71d" containerName="payload-processing" Sep 07 13:26:31.045530 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.045477 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="be0df32e-8334-442f-9460-46bb6b2e4923" containerName="revoke-keys" Sep 07 13:26:31.045530 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.045489 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5042b6d7-912b-4ef1-ace2-5446a290c71d" containerName="payload-processing" Sep 07 13:26:31.049888 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.049871 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp" Sep 07 13:26:31.052242 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.052212 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 13:26:31.052363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.052294 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jlvcf\"" Sep 07 13:26:31.058578 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.058547 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp"] Sep 07 13:26:31.159802 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.159771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1944c56c-9034-43fc-bd5d-9b418b08f1cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp\" (UID: \"1944c56c-9034-43fc-bd5d-9b418b08f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp" Sep 07 13:26:31.260800 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.260758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1944c56c-9034-43fc-bd5d-9b418b08f1cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp\" (UID: \"1944c56c-9034-43fc-bd5d-9b418b08f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp" Sep 07 13:26:31.261413 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.261393 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1944c56c-9034-43fc-bd5d-9b418b08f1cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp\" (UID: \"1944c56c-9034-43fc-bd5d-9b418b08f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp" Sep 07 13:26:31.361873 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.361759 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp" Sep 07 13:26:31.653120 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.653090 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp"] Sep 07 13:26:31.654122 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:26:31.654094 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1944c56c_9034_43fc_bd5d_9b418b08f1cc.slice/crio-3ed4c0f797ff4b824d5b424f343558d8e0f349754f630ca8438cc5f3b0b19cd3 WatchSource:0}: Error finding container 3ed4c0f797ff4b824d5b424f343558d8e0f349754f630ca8438cc5f3b0b19cd3: Status 404 returned error can't find the container with id 3ed4c0f797ff4b824d5b424f343558d8e0f349754f630ca8438cc5f3b0b19cd3 Sep 07 13:26:31.694720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:31.694689 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp" event={"ID":"1944c56c-9034-43fc-bd5d-9b418b08f1cc","Type":"ContainerStarted","Data":"3ed4c0f797ff4b824d5b424f343558d8e0f349754f630ca8438cc5f3b0b19cd3"} Sep 07 13:26:32.701989 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:32.701951 2565 generic.go:356] "Generic (PLEG): container finished" podID="1944c56c-9034-43fc-bd5d-9b418b08f1cc" containerID="12ff6496b0ff6b8b7779be7ec91334d7e9c5ca10a2c7b6bc0d66ae82ff229eff" exitCode=0 Sep 07 13:26:32.702362 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:32.702005 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp" event={"ID":"1944c56c-9034-43fc-bd5d-9b418b08f1cc","Type":"ContainerDied","Data":"12ff6496b0ff6b8b7779be7ec91334d7e9c5ca10a2c7b6bc0d66ae82ff229eff"} Sep 07 13:26:33.844089 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:33.844062 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp" Sep 07 13:26:33.988463 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:33.988370 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1944c56c-9034-43fc-bd5d-9b418b08f1cc-maas-api-service-ca\") pod \"1944c56c-9034-43fc-bd5d-9b418b08f1cc\" (UID: \"1944c56c-9034-43fc-bd5d-9b418b08f1cc\") " Sep 07 13:26:33.988759 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:33.988737 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1944c56c-9034-43fc-bd5d-9b418b08f1cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1944c56c-9034-43fc-bd5d-9b418b08f1cc" (UID: "1944c56c-9034-43fc-bd5d-9b418b08f1cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:26:34.089665 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:34.089636 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1944c56c-9034-43fc-bd5d-9b418b08f1cc-maas-api-service-ca\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:34.713234 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:34.713196 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp" event={"ID":"1944c56c-9034-43fc-bd5d-9b418b08f1cc","Type":"ContainerDied","Data":"3ed4c0f797ff4b824d5b424f343558d8e0f349754f630ca8438cc5f3b0b19cd3"} Sep 07 13:26:34.713234 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:34.713224 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp" Sep 07 13:26:34.713441 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:34.713256 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ed4c0f797ff4b824d5b424f343558d8e0f349754f630ca8438cc5f3b0b19cd3" Sep 07 13:26:35.140556 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:35.140482 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq_334fe76b-ad10-4ff5-b64a-da7ba597b1b4/payload-processing/0.log" Sep 07 13:26:41.744267 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:41.744228 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq"] Sep 07 13:26:41.744785 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:41.744549 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" podUID="d20267fc-bec6-4bf4-8bfc-62388088fac7" containerName="payload-processing" containerID="cri-o://a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba" gracePeriod=30 Sep 07 13:26:41.753723 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:41.753698 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd"] Sep 07 13:26:41.753944 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:41.753901 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" podUID="3786fb7e-0ac3-4873-95d1-6b4fb7bab98d" containerName="payload-pre-processing" containerID="cri-o://1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf" gracePeriod=30 Sep 07 13:26:42.033872 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.033840 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:26:42.035613 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.035568 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp"] Sep 07 13:26:42.037817 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.037751 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:26:42.039777 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.039759 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8e7ddc-56gsp"] Sep 07 13:26:42.166765 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.166738 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-plugins-config-volume\") pod \"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d\" (UID: \"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d\") " Sep 07 13:26:42.166917 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.166799 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgsjl\" (UniqueName: \"kubernetes.io/projected/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-kube-api-access-hgsjl\") pod \"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d\" (UID: \"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d\") " Sep 07 13:26:42.166917 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.166855 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxhbl\" (UniqueName: \"kubernetes.io/projected/d20267fc-bec6-4bf4-8bfc-62388088fac7-kube-api-access-pxhbl\") pod \"d20267fc-bec6-4bf4-8bfc-62388088fac7\" (UID: \"d20267fc-bec6-4bf4-8bfc-62388088fac7\") " Sep 07 13:26:42.166917 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.166902 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20267fc-bec6-4bf4-8bfc-62388088fac7-plugins-config-volume\") pod \"d20267fc-bec6-4bf4-8bfc-62388088fac7\" (UID: \"d20267fc-bec6-4bf4-8bfc-62388088fac7\") " Sep 07 13:26:42.167361 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.167330 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3786fb7e-0ac3-4873-95d1-6b4fb7bab98d" (UID: "3786fb7e-0ac3-4873-95d1-6b4fb7bab98d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:26:42.167455 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.167343 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20267fc-bec6-4bf4-8bfc-62388088fac7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d20267fc-bec6-4bf4-8bfc-62388088fac7" (UID: "d20267fc-bec6-4bf4-8bfc-62388088fac7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:26:42.169404 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.169283 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20267fc-bec6-4bf4-8bfc-62388088fac7-kube-api-access-pxhbl" (OuterVolumeSpecName: "kube-api-access-pxhbl") pod "d20267fc-bec6-4bf4-8bfc-62388088fac7" (UID: "d20267fc-bec6-4bf4-8bfc-62388088fac7"). InnerVolumeSpecName "kube-api-access-pxhbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:26:42.170540 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.170520 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-kube-api-access-hgsjl" (OuterVolumeSpecName: "kube-api-access-hgsjl") pod "3786fb7e-0ac3-4873-95d1-6b4fb7bab98d" (UID: "3786fb7e-0ac3-4873-95d1-6b4fb7bab98d"). InnerVolumeSpecName "kube-api-access-hgsjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:26:42.268522 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.268445 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxhbl\" (UniqueName: \"kubernetes.io/projected/d20267fc-bec6-4bf4-8bfc-62388088fac7-kube-api-access-pxhbl\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:42.268522 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.268475 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20267fc-bec6-4bf4-8bfc-62388088fac7-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:42.268522 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.268484 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:42.268522 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.268495 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgsjl\" (UniqueName: \"kubernetes.io/projected/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d-kube-api-access-hgsjl\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:26:42.755081 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.755043 2565 generic.go:356] "Generic (PLEG): container finished" podID="d20267fc-bec6-4bf4-8bfc-62388088fac7" containerID="a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba" exitCode=0 Sep 07 13:26:42.755512 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.755090 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" Sep 07 13:26:42.755512 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.755131 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" event={"ID":"d20267fc-bec6-4bf4-8bfc-62388088fac7","Type":"ContainerDied","Data":"a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba"} Sep 07 13:26:42.755512 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.755185 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq" event={"ID":"d20267fc-bec6-4bf4-8bfc-62388088fac7","Type":"ContainerDied","Data":"76f43cfe680f4f4733132d56ddb340546847326f21091c13648d335eec953e6c"} Sep 07 13:26:42.755512 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.755207 2565 scope.go:117] "RemoveContainer" containerID="a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba" Sep 07 13:26:42.756623 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.756546 2565 generic.go:356] "Generic (PLEG): container finished" podID="3786fb7e-0ac3-4873-95d1-6b4fb7bab98d" containerID="1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf" exitCode=0 Sep 07 13:26:42.756623 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.756578 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" event={"ID":"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d","Type":"ContainerDied","Data":"1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf"} Sep 07 13:26:42.756732 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.756645 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" event={"ID":"3786fb7e-0ac3-4873-95d1-6b4fb7bab98d","Type":"ContainerDied","Data":"181ac2bb03b98829743d75c8c8c77c985e3713e41c70bcea496492dc08c2ec04"} Sep 07 13:26:42.756732 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.756582 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd" Sep 07 13:26:42.767339 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.767043 2565 scope.go:117] "RemoveContainer" containerID="a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba" Sep 07 13:26:42.767702 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:26:42.767664 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba\": container with ID starting with a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba not found: ID does not exist" containerID="a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba" Sep 07 13:26:42.767791 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.767714 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba"} err="failed to get container status \"a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba\": rpc error: code = NotFound desc = could not find container \"a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba\": container with ID starting with a921d9068c5d5a13f3602efab717dcd3ed68061cd273bff97ca40fa9feab04ba not found: ID does not exist" Sep 07 13:26:42.767791 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.767736 2565 scope.go:117] "RemoveContainer" containerID="1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf" Sep 07 13:26:42.776398 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.776370 2565 scope.go:117] "RemoveContainer" containerID="1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf" Sep 07 13:26:42.776678 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:26:42.776655 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf\": container with ID starting with 1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf not found: ID does not exist" containerID="1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf" Sep 07 13:26:42.776729 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.776688 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf"} err="failed to get container status \"1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf\": rpc error: code = NotFound desc = could not find container \"1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf\": container with ID starting with 1e154188336652e5b8c73fd4b2d4ddb223621c21c6d872db64f6aafb23edf9bf not found: ID does not exist" Sep 07 13:26:42.794977 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.794955 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd"] Sep 07 13:26:42.802813 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.802792 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8e7ddc-7c8548dcb6-69jbd"] Sep 07 13:26:42.827522 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.827493 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1944c56c-9034-43fc-bd5d-9b418b08f1cc" path="/var/lib/kubelet/pods/1944c56c-9034-43fc-bd5d-9b418b08f1cc/volumes" Sep 07 13:26:42.828256 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.828065 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3786fb7e-0ac3-4873-95d1-6b4fb7bab98d" path="/var/lib/kubelet/pods/3786fb7e-0ac3-4873-95d1-6b4fb7bab98d/volumes" Sep 07 13:26:42.828471 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.828453 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq"] Sep 07 13:26:42.829853 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:42.829831 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8e7ddc-5c8dd5bd8f-krzdq"] Sep 07 13:26:43.260333 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.260300 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96"] Sep 07 13:26:43.260760 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.260745 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1944c56c-9034-43fc-bd5d-9b418b08f1cc" containerName="revoke-keys" Sep 07 13:26:43.260813 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.260762 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1944c56c-9034-43fc-bd5d-9b418b08f1cc" containerName="revoke-keys" Sep 07 13:26:43.260813 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.260777 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3786fb7e-0ac3-4873-95d1-6b4fb7bab98d" containerName="payload-pre-processing" Sep 07 13:26:43.260813 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.260784 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3786fb7e-0ac3-4873-95d1-6b4fb7bab98d" containerName="payload-pre-processing" Sep 07 13:26:43.260813 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.260798 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20267fc-bec6-4bf4-8bfc-62388088fac7" containerName="payload-processing" Sep 07 13:26:43.260813 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.260807 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20267fc-bec6-4bf4-8bfc-62388088fac7" containerName="payload-processing" Sep 07 13:26:43.260967 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.260866 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20267fc-bec6-4bf4-8bfc-62388088fac7" containerName="payload-processing" Sep 07 13:26:43.260967 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.260874 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1944c56c-9034-43fc-bd5d-9b418b08f1cc" containerName="revoke-keys" Sep 07 13:26:43.260967 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.260884 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3786fb7e-0ac3-4873-95d1-6b4fb7bab98d" containerName="payload-pre-processing" Sep 07 13:26:43.265308 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.265292 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.269433 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.269410 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-219f63\"/\"openshift-service-ca.crt\"" Sep 07 13:26:43.269433 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.269411 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-219f63\"/\"ipp-route-219f63-kserve-self-signed-certs\"" Sep 07 13:26:43.270265 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.270248 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-219f63\"/\"kube-root-ca.crt\"" Sep 07 13:26:43.270358 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.270317 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-219f63\"/\"default-dockercfg-bk8j8\"" Sep 07 13:26:43.277675 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.277655 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96"] Sep 07 13:26:43.379145 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.379105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-dshm\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.379337 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.379161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tls-certs\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.379337 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.379254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v8ch\" (UniqueName: \"kubernetes.io/projected/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-kube-api-access-2v8ch\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.379485 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.379340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-home\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.379485 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.379379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tmp-dir\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.379485 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.379414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-model-cache\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.480564 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.480527 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v8ch\" (UniqueName: \"kubernetes.io/projected/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-kube-api-access-2v8ch\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.480748 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.480585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-home\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.480748 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.480655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tmp-dir\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.480748 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.480690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-model-cache\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.480906 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.480771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-dshm\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.480973 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.480911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tls-certs\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.481052 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.481030 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-home\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.481115 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.481041 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tmp-dir\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.481173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.481129 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-model-cache\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.482903 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.482886 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-dshm\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.483217 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.483200 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tls-certs\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.499099 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.499073 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v8ch\" (UniqueName: \"kubernetes.io/projected/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-kube-api-access-2v8ch\") pod \"ipp-route-219f63-kserve-bb9d8c895-psg96\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.574814 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.574725 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:43.874241 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:43.874190 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96"] Sep 07 13:26:43.876549 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:26:43.876522 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bfca2ac_dee4_48a7_a6cc_35933cd3a1fd.slice/crio-88353ae36584920ba708aa91f0c51858708ffe79add02f9bd65e80488a34417a WatchSource:0}: Error finding container 88353ae36584920ba708aa91f0c51858708ffe79add02f9bd65e80488a34417a: Status 404 returned error can't find the container with id 88353ae36584920ba708aa91f0c51858708ffe79add02f9bd65e80488a34417a Sep 07 13:26:44.771666 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:44.771625 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" event={"ID":"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd","Type":"ContainerStarted","Data":"dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1"} Sep 07 13:26:44.771666 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:44.771671 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" event={"ID":"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd","Type":"ContainerStarted","Data":"88353ae36584920ba708aa91f0c51858708ffe79add02f9bd65e80488a34417a"} Sep 07 13:26:44.796005 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:44.795960 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" podStartSLOduration=1.7959469879999999 podStartE2EDuration="1.795946988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:26:44.794570707 +0000 UTC m=+1626.474956478" watchObservedRunningTime="2026-09-07 13:26:44.795946988 +0000 UTC m=+1626.476332767" Sep 07 13:26:44.828642 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:44.828612 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20267fc-bec6-4bf4-8bfc-62388088fac7" path="/var/lib/kubelet/pods/d20267fc-bec6-4bf4-8bfc-62388088fac7/volumes" Sep 07 13:26:53.575810 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:53.575780 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:53.576269 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:53.576102 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:53.588558 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:53.588535 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:26:53.833057 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:26:53.832977 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:27:11.103708 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:11.103679 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq_334fe76b-ad10-4ff5-b64a-da7ba597b1b4/payload-processing/0.log" Sep 07 13:27:16.937068 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:16.937032 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl"] Sep 07 13:27:16.940855 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:16.940833 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:16.943900 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:16.943882 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d14aa2\"" Sep 07 13:27:16.944215 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:16.944201 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d14aa2-dockercfg-qc7zv\"" Sep 07 13:27:16.952940 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:16.952918 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl"] Sep 07 13:27:17.089228 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.089192 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfj7q\" (UniqueName: \"kubernetes.io/projected/6812db13-ba49-4375-bcc0-743e30f7302b-kube-api-access-jfj7q\") pod \"payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl\" (UID: \"6812db13-ba49-4375-bcc0-743e30f7302b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:17.089228 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.089234 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6812db13-ba49-4375-bcc0-743e30f7302b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl\" (UID: \"6812db13-ba49-4375-bcc0-743e30f7302b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:17.190908 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.190832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfj7q\" (UniqueName: \"kubernetes.io/projected/6812db13-ba49-4375-bcc0-743e30f7302b-kube-api-access-jfj7q\") pod \"payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl\" (UID: \"6812db13-ba49-4375-bcc0-743e30f7302b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:17.190908 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.190868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6812db13-ba49-4375-bcc0-743e30f7302b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl\" (UID: \"6812db13-ba49-4375-bcc0-743e30f7302b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:17.191498 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.191478 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6812db13-ba49-4375-bcc0-743e30f7302b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl\" (UID: \"6812db13-ba49-4375-bcc0-743e30f7302b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:17.210959 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.210922 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfj7q\" (UniqueName: \"kubernetes.io/projected/6812db13-ba49-4375-bcc0-743e30f7302b-kube-api-access-jfj7q\") pod \"payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl\" (UID: \"6812db13-ba49-4375-bcc0-743e30f7302b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:17.252549 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.252520 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:17.554663 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:27:17.554576 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6812db13_ba49_4375_bcc0_743e30f7302b.slice/crio-6ecba811278c0abd96363388bc415dc93b52ae03ba0440caeaa9d0a62637bec2 WatchSource:0}: Error finding container 6ecba811278c0abd96363388bc415dc93b52ae03ba0440caeaa9d0a62637bec2: Status 404 returned error can't find the container with id 6ecba811278c0abd96363388bc415dc93b52ae03ba0440caeaa9d0a62637bec2 Sep 07 13:27:17.555640 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.555618 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl"] Sep 07 13:27:17.947491 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.947457 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" event={"ID":"6812db13-ba49-4375-bcc0-743e30f7302b","Type":"ContainerStarted","Data":"022283238e73a25612748bfc9538a9ee29c3ea89b4afd8c451476eefb0d5b2ec"} Sep 07 13:27:17.947865 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.947517 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:17.947865 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.947540 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" event={"ID":"6812db13-ba49-4375-bcc0-743e30f7302b","Type":"ContainerStarted","Data":"6ecba811278c0abd96363388bc415dc93b52ae03ba0440caeaa9d0a62637bec2"} Sep 07 13:27:17.982310 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:17.982262 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" podStartSLOduration=1.982248229 podStartE2EDuration="1.982248229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:27:17.981555302 +0000 UTC m=+1659.661941071" watchObservedRunningTime="2026-09-07 13:27:17.982248229 +0000 UTC m=+1659.662633998" Sep 07 13:27:28.955695 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:28.955660 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:40.575342 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:40.575299 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd"] Sep 07 13:27:40.580338 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:40.580307 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd" Sep 07 13:27:40.587364 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:40.587344 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jlvcf\"" Sep 07 13:27:40.587517 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:40.587499 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 13:27:40.600561 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:40.600536 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd"] Sep 07 13:27:40.605619 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:40.605573 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e3a2a5-dcbc-47ec-9585-d99eb15f5697-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd\" (UID: \"73e3a2a5-dcbc-47ec-9585-d99eb15f5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd" Sep 07 13:27:40.706543 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:40.706513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e3a2a5-dcbc-47ec-9585-d99eb15f5697-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd\" (UID: \"73e3a2a5-dcbc-47ec-9585-d99eb15f5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd" Sep 07 13:27:40.707145 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:40.707121 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e3a2a5-dcbc-47ec-9585-d99eb15f5697-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd\" (UID: \"73e3a2a5-dcbc-47ec-9585-d99eb15f5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd" Sep 07 13:27:40.890351 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:40.890272 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd" Sep 07 13:27:41.169637 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:41.169612 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd"] Sep 07 13:27:41.171398 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:27:41.171369 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e3a2a5_dcbc_47ec_9585_d99eb15f5697.slice/crio-55e81369188f0499898fb18ed79c1df0aea37df49f7c841f2b24eadaf2c5be86 WatchSource:0}: Error finding container 55e81369188f0499898fb18ed79c1df0aea37df49f7c841f2b24eadaf2c5be86: Status 404 returned error can't find the container with id 55e81369188f0499898fb18ed79c1df0aea37df49f7c841f2b24eadaf2c5be86 Sep 07 13:27:42.077650 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:42.077585 2565 generic.go:356] "Generic (PLEG): container finished" podID="73e3a2a5-dcbc-47ec-9585-d99eb15f5697" containerID="68921f916be81a2de8d207624e9bc325c32b0b96346e89e8f8b0cff4078d0037" exitCode=0 Sep 07 13:27:42.078024 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:42.077671 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd" event={"ID":"73e3a2a5-dcbc-47ec-9585-d99eb15f5697","Type":"ContainerDied","Data":"68921f916be81a2de8d207624e9bc325c32b0b96346e89e8f8b0cff4078d0037"} Sep 07 13:27:42.078024 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:42.077761 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd" event={"ID":"73e3a2a5-dcbc-47ec-9585-d99eb15f5697","Type":"ContainerStarted","Data":"55e81369188f0499898fb18ed79c1df0aea37df49f7c841f2b24eadaf2c5be86"} Sep 07 13:27:43.219844 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:43.219819 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd" Sep 07 13:27:43.336860 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:43.336792 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e3a2a5-dcbc-47ec-9585-d99eb15f5697-maas-api-service-ca\") pod \"73e3a2a5-dcbc-47ec-9585-d99eb15f5697\" (UID: \"73e3a2a5-dcbc-47ec-9585-d99eb15f5697\") " Sep 07 13:27:43.337214 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:43.337186 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e3a2a5-dcbc-47ec-9585-d99eb15f5697-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73e3a2a5-dcbc-47ec-9585-d99eb15f5697" (UID: "73e3a2a5-dcbc-47ec-9585-d99eb15f5697"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:27:43.438590 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:43.438554 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e3a2a5-dcbc-47ec-9585-d99eb15f5697-maas-api-service-ca\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:27:44.089861 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:44.089829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd" event={"ID":"73e3a2a5-dcbc-47ec-9585-d99eb15f5697","Type":"ContainerDied","Data":"55e81369188f0499898fb18ed79c1df0aea37df49f7c841f2b24eadaf2c5be86"} Sep 07 13:27:44.090073 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:44.089866 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd" Sep 07 13:27:44.090073 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:44.089873 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55e81369188f0499898fb18ed79c1df0aea37df49f7c841f2b24eadaf2c5be86" Sep 07 13:27:46.967827 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:46.967796 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl"] Sep 07 13:27:46.968291 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:46.968147 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" podUID="6812db13-ba49-4375-bcc0-743e30f7302b" containerName="payload-pre-processing" containerID="cri-o://022283238e73a25612748bfc9538a9ee29c3ea89b4afd8c451476eefb0d5b2ec" gracePeriod=30 Sep 07 13:27:47.108113 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:47.108082 2565 generic.go:356] "Generic (PLEG): container finished" podID="6812db13-ba49-4375-bcc0-743e30f7302b" containerID="022283238e73a25612748bfc9538a9ee29c3ea89b4afd8c451476eefb0d5b2ec" exitCode=0 Sep 07 13:27:47.108291 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:47.108150 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" event={"ID":"6812db13-ba49-4375-bcc0-743e30f7302b","Type":"ContainerDied","Data":"022283238e73a25612748bfc9538a9ee29c3ea89b4afd8c451476eefb0d5b2ec"} Sep 07 13:27:47.233298 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:47.233243 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:47.373538 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:47.373505 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6812db13-ba49-4375-bcc0-743e30f7302b-plugins-config-volume\") pod \"6812db13-ba49-4375-bcc0-743e30f7302b\" (UID: \"6812db13-ba49-4375-bcc0-743e30f7302b\") " Sep 07 13:27:47.373701 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:47.373553 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfj7q\" (UniqueName: \"kubernetes.io/projected/6812db13-ba49-4375-bcc0-743e30f7302b-kube-api-access-jfj7q\") pod \"6812db13-ba49-4375-bcc0-743e30f7302b\" (UID: \"6812db13-ba49-4375-bcc0-743e30f7302b\") " Sep 07 13:27:47.373953 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:47.373926 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6812db13-ba49-4375-bcc0-743e30f7302b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6812db13-ba49-4375-bcc0-743e30f7302b" (UID: "6812db13-ba49-4375-bcc0-743e30f7302b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:27:47.375658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:47.375626 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6812db13-ba49-4375-bcc0-743e30f7302b-kube-api-access-jfj7q" (OuterVolumeSpecName: "kube-api-access-jfj7q") pod "6812db13-ba49-4375-bcc0-743e30f7302b" (UID: "6812db13-ba49-4375-bcc0-743e30f7302b"). InnerVolumeSpecName "kube-api-access-jfj7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:27:47.422320 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:47.422290 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd"] Sep 07 13:27:47.427667 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:47.427643 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d14aa2-j68hd"] Sep 07 13:27:47.474894 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:47.474864 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfj7q\" (UniqueName: \"kubernetes.io/projected/6812db13-ba49-4375-bcc0-743e30f7302b-kube-api-access-jfj7q\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:27:47.474894 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:47.474891 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6812db13-ba49-4375-bcc0-743e30f7302b-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:27:48.115239 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:48.115204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" event={"ID":"6812db13-ba49-4375-bcc0-743e30f7302b","Type":"ContainerDied","Data":"6ecba811278c0abd96363388bc415dc93b52ae03ba0440caeaa9d0a62637bec2"} Sep 07 13:27:48.115239 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:48.115226 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl" Sep 07 13:27:48.115792 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:48.115264 2565 scope.go:117] "RemoveContainer" containerID="022283238e73a25612748bfc9538a9ee29c3ea89b4afd8c451476eefb0d5b2ec" Sep 07 13:27:48.140775 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:48.140751 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl"] Sep 07 13:27:48.143153 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:48.143131 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d14aa2-69449dfff8-krcwl"] Sep 07 13:27:48.828041 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:48.828008 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6812db13-ba49-4375-bcc0-743e30f7302b" path="/var/lib/kubelet/pods/6812db13-ba49-4375-bcc0-743e30f7302b/volumes" Sep 07 13:27:48.828512 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:48.828494 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73e3a2a5-dcbc-47ec-9585-d99eb15f5697" path="/var/lib/kubelet/pods/73e3a2a5-dcbc-47ec-9585-d99eb15f5697/volumes" Sep 07 13:27:58.155182 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.155147 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r"] Sep 07 13:27:58.155630 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.155613 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6812db13-ba49-4375-bcc0-743e30f7302b" containerName="payload-pre-processing" Sep 07 13:27:58.155682 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.155631 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6812db13-ba49-4375-bcc0-743e30f7302b" containerName="payload-pre-processing" Sep 07 13:27:58.155682 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.155641 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73e3a2a5-dcbc-47ec-9585-d99eb15f5697" containerName="revoke-keys" Sep 07 13:27:58.155682 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.155647 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="73e3a2a5-dcbc-47ec-9585-d99eb15f5697" containerName="revoke-keys" Sep 07 13:27:58.155777 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.155721 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="73e3a2a5-dcbc-47ec-9585-d99eb15f5697" containerName="revoke-keys" Sep 07 13:27:58.155777 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.155735 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6812db13-ba49-4375-bcc0-743e30f7302b" containerName="payload-pre-processing" Sep 07 13:27:58.160126 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.160107 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" Sep 07 13:27:58.164363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.164342 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jlvcf\"" Sep 07 13:27:58.164769 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.164753 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 13:27:58.181095 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.181072 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r"] Sep 07 13:27:58.271555 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.271527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df31c49-cb9c-4dcb-a2d4-284c6eea5e40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r\" (UID: \"4df31c49-cb9c-4dcb-a2d4-284c6eea5e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" Sep 07 13:27:58.372950 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.372922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df31c49-cb9c-4dcb-a2d4-284c6eea5e40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r\" (UID: \"4df31c49-cb9c-4dcb-a2d4-284c6eea5e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" Sep 07 13:27:58.373486 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.373467 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df31c49-cb9c-4dcb-a2d4-284c6eea5e40-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r\" (UID: \"4df31c49-cb9c-4dcb-a2d4-284c6eea5e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" Sep 07 13:27:58.470459 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.470395 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" Sep 07 13:27:58.745034 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:58.745003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r"] Sep 07 13:27:58.747028 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:27:58.747004 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df31c49_cb9c_4dcb_a2d4_284c6eea5e40.slice/crio-9cab90e89e2c6144c1168258c3e8a69fcc9213df4b015e65947507342c9bc15e WatchSource:0}: Error finding container 9cab90e89e2c6144c1168258c3e8a69fcc9213df4b015e65947507342c9bc15e: Status 404 returned error can't find the container with id 9cab90e89e2c6144c1168258c3e8a69fcc9213df4b015e65947507342c9bc15e Sep 07 13:27:59.172206 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:59.172170 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" event={"ID":"4df31c49-cb9c-4dcb-a2d4-284c6eea5e40","Type":"ContainerStarted","Data":"bca0c1c38477f52574dcb8feecafbad4d72f76f81e08014986c23179cfd0626c"} Sep 07 13:27:59.172537 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:59.172226 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" event={"ID":"4df31c49-cb9c-4dcb-a2d4-284c6eea5e40","Type":"ContainerStarted","Data":"9cab90e89e2c6144c1168258c3e8a69fcc9213df4b015e65947507342c9bc15e"} Sep 07 13:27:59.197569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:27:59.197521 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" podStartSLOduration=1.197507505 podStartE2EDuration="1.197507505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:27:59.195550262 +0000 UTC m=+1700.875936035" watchObservedRunningTime="2026-09-07 13:27:59.197507505 +0000 UTC m=+1700.877893274" Sep 07 13:28:00.179186 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:00.179150 2565 generic.go:356] "Generic (PLEG): container finished" podID="4df31c49-cb9c-4dcb-a2d4-284c6eea5e40" containerID="bca0c1c38477f52574dcb8feecafbad4d72f76f81e08014986c23179cfd0626c" exitCode=0 Sep 07 13:28:00.179653 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:00.179233 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" event={"ID":"4df31c49-cb9c-4dcb-a2d4-284c6eea5e40","Type":"ContainerDied","Data":"bca0c1c38477f52574dcb8feecafbad4d72f76f81e08014986c23179cfd0626c"} Sep 07 13:28:01.316023 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:01.316000 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" Sep 07 13:28:01.398909 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:01.398847 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df31c49-cb9c-4dcb-a2d4-284c6eea5e40-maas-api-service-ca\") pod \"4df31c49-cb9c-4dcb-a2d4-284c6eea5e40\" (UID: \"4df31c49-cb9c-4dcb-a2d4-284c6eea5e40\") " Sep 07 13:28:01.399195 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:01.399162 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df31c49-cb9c-4dcb-a2d4-284c6eea5e40-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4df31c49-cb9c-4dcb-a2d4-284c6eea5e40" (UID: "4df31c49-cb9c-4dcb-a2d4-284c6eea5e40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:28:01.500360 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:01.500297 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4df31c49-cb9c-4dcb-a2d4-284c6eea5e40-maas-api-service-ca\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:28:02.191586 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:02.191548 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" event={"ID":"4df31c49-cb9c-4dcb-a2d4-284c6eea5e40","Type":"ContainerDied","Data":"9cab90e89e2c6144c1168258c3e8a69fcc9213df4b015e65947507342c9bc15e"} Sep 07 13:28:02.191586 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:02.191588 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r" Sep 07 13:28:02.191868 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:02.191619 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cab90e89e2c6144c1168258c3e8a69fcc9213df4b015e65947507342c9bc15e" Sep 07 13:28:13.941470 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:13.941418 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5"] Sep 07 13:28:13.941986 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:13.941678 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" podUID="d55f934f-38bb-4ea8-b721-874ab9845cdb" containerName="payload-pre-processing" containerID="cri-o://9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703" gracePeriod=30 Sep 07 13:28:14.202871 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.202811 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:28:14.215581 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.215558 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55f934f-38bb-4ea8-b721-874ab9845cdb-plugins-config-volume\") pod \"d55f934f-38bb-4ea8-b721-874ab9845cdb\" (UID: \"d55f934f-38bb-4ea8-b721-874ab9845cdb\") " Sep 07 13:28:14.215727 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.215611 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzs52\" (UniqueName: \"kubernetes.io/projected/d55f934f-38bb-4ea8-b721-874ab9845cdb-kube-api-access-bzs52\") pod \"d55f934f-38bb-4ea8-b721-874ab9845cdb\" (UID: \"d55f934f-38bb-4ea8-b721-874ab9845cdb\") " Sep 07 13:28:14.215899 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.215876 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55f934f-38bb-4ea8-b721-874ab9845cdb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d55f934f-38bb-4ea8-b721-874ab9845cdb" (UID: "d55f934f-38bb-4ea8-b721-874ab9845cdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:28:14.217567 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.217543 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55f934f-38bb-4ea8-b721-874ab9845cdb-kube-api-access-bzs52" (OuterVolumeSpecName: "kube-api-access-bzs52") pod "d55f934f-38bb-4ea8-b721-874ab9845cdb" (UID: "d55f934f-38bb-4ea8-b721-874ab9845cdb"). InnerVolumeSpecName "kube-api-access-bzs52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:28:14.237263 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.237239 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r"] Sep 07 13:28:14.241895 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.241872 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-219f63-vn56r"] Sep 07 13:28:14.254778 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.254747 2565 generic.go:356] "Generic (PLEG): container finished" podID="d55f934f-38bb-4ea8-b721-874ab9845cdb" containerID="9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703" exitCode=0 Sep 07 13:28:14.254908 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.254786 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" Sep 07 13:28:14.254908 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.254789 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" event={"ID":"d55f934f-38bb-4ea8-b721-874ab9845cdb","Type":"ContainerDied","Data":"9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703"} Sep 07 13:28:14.255030 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.254910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5" event={"ID":"d55f934f-38bb-4ea8-b721-874ab9845cdb","Type":"ContainerDied","Data":"980fa11b5ea1bdeed9b6ce23516a2a768f22c6962a87fe0a5bfb1f2381f4039d"} Sep 07 13:28:14.255030 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.254945 2565 scope.go:117] "RemoveContainer" containerID="9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703" Sep 07 13:28:14.263860 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.263845 2565 scope.go:117] "RemoveContainer" containerID="9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703" Sep 07 13:28:14.264081 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:28:14.264059 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703\": container with ID starting with 9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703 not found: ID does not exist" containerID="9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703" Sep 07 13:28:14.264139 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.264089 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703"} err="failed to get container status \"9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703\": rpc error: code = NotFound desc = could not find container \"9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703\": container with ID starting with 9cbbef28dc5e3f6d4bed7c1283bf28894007d3c0e2f08e64fa408fb7ecaed703 not found: ID does not exist" Sep 07 13:28:14.281565 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.281544 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5"] Sep 07 13:28:14.286722 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.286702 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219f63-6975f8fdd6-cdpz5"] Sep 07 13:28:14.316316 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.316294 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d55f934f-38bb-4ea8-b721-874ab9845cdb-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:28:14.316316 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.316314 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzs52\" (UniqueName: \"kubernetes.io/projected/d55f934f-38bb-4ea8-b721-874ab9845cdb-kube-api-access-bzs52\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:28:14.826447 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.826409 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df31c49-cb9c-4dcb-a2d4-284c6eea5e40" path="/var/lib/kubelet/pods/4df31c49-cb9c-4dcb-a2d4-284c6eea5e40/volumes" Sep 07 13:28:14.826839 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:14.826824 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d55f934f-38bb-4ea8-b721-874ab9845cdb" path="/var/lib/kubelet/pods/d55f934f-38bb-4ea8-b721-874ab9845cdb/volumes" Sep 07 13:28:19.582090 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:19.582049 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96"] Sep 07 13:28:34.602485 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.602420 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" podUID="7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" containerName="main" containerID="cri-o://dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1" gracePeriod=45 Sep 07 13:28:34.853533 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.853473 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:28:34.905174 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905141 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-model-cache\") pod \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " Sep 07 13:28:34.905350 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905264 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-home\") pod \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " Sep 07 13:28:34.905350 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905302 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v8ch\" (UniqueName: \"kubernetes.io/projected/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-kube-api-access-2v8ch\") pod \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " Sep 07 13:28:34.905350 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905334 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-dshm\") pod \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " Sep 07 13:28:34.905523 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905377 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tls-certs\") pod \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " Sep 07 13:28:34.905523 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905401 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tmp-dir\") pod \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\" (UID: \"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd\") " Sep 07 13:28:34.905523 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905415 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" (UID: "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:28:34.905720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905556 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-home" (OuterVolumeSpecName: "home") pod "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" (UID: "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:28:34.905720 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905697 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" (UID: "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:28:34.906217 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905901 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-home\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:28:34.906217 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905926 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tmp-dir\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:28:34.906217 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.905941 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-model-cache\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:28:34.908093 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.908065 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" (UID: "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:28:34.908198 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.908133 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-dshm" (OuterVolumeSpecName: "dshm") pod "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" (UID: "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:28:34.908245 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:34.908209 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-kube-api-access-2v8ch" (OuterVolumeSpecName: "kube-api-access-2v8ch") pod "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" (UID: "7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd"). InnerVolumeSpecName "kube-api-access-2v8ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:28:35.006688 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.006648 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v8ch\" (UniqueName: \"kubernetes.io/projected/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-kube-api-access-2v8ch\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:28:35.006688 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.006683 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-dshm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:28:35.006688 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.006694 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd-tls-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:28:35.362668 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.362635 2565 generic.go:356] "Generic (PLEG): container finished" podID="7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" containerID="dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1" exitCode=0 Sep 07 13:28:35.362843 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.362677 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" Sep 07 13:28:35.362843 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.362722 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" event={"ID":"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd","Type":"ContainerDied","Data":"dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1"} Sep 07 13:28:35.362843 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.362763 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96" event={"ID":"7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd","Type":"ContainerDied","Data":"88353ae36584920ba708aa91f0c51858708ffe79add02f9bd65e80488a34417a"} Sep 07 13:28:35.362843 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.362780 2565 scope.go:117] "RemoveContainer" containerID="dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1" Sep 07 13:28:35.371966 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.371949 2565 scope.go:117] "RemoveContainer" containerID="dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1" Sep 07 13:28:35.372215 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:28:35.372195 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1\": container with ID starting with dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1 not found: ID does not exist" containerID="dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1" Sep 07 13:28:35.372272 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.372223 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1"} err="failed to get container status \"dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1\": rpc error: code = NotFound desc = could not find container \"dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1\": container with ID starting with dc36942d00163b0d2b5be68b1be91a8e6d2059e64df4c324ca8d308696e4bfa1 not found: ID does not exist" Sep 07 13:28:35.387221 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.387200 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96"] Sep 07 13:28:35.391119 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:35.391099 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-219f63/ipp-route-219f63-kserve-bb9d8c895-psg96"] Sep 07 13:28:36.826805 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:28:36.826771 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" path="/var/lib/kubelet/pods/7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd/volumes" Sep 07 13:29:03.762751 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.762711 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr"] Sep 07 13:29:03.763313 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.763295 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55f934f-38bb-4ea8-b721-874ab9845cdb" containerName="payload-pre-processing" Sep 07 13:29:03.763363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.763315 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55f934f-38bb-4ea8-b721-874ab9845cdb" containerName="payload-pre-processing" Sep 07 13:29:03.763363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.763330 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df31c49-cb9c-4dcb-a2d4-284c6eea5e40" containerName="revoke-keys" Sep 07 13:29:03.763363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.763337 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df31c49-cb9c-4dcb-a2d4-284c6eea5e40" containerName="revoke-keys" Sep 07 13:29:03.763363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.763356 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" containerName="main" Sep 07 13:29:03.763363 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.763361 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" containerName="main" Sep 07 13:29:03.763525 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.763417 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bfca2ac-dee4-48a7-a6cc-35933cd3a1fd" containerName="main" Sep 07 13:29:03.763525 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.763427 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df31c49-cb9c-4dcb-a2d4-284c6eea5e40" containerName="revoke-keys" Sep 07 13:29:03.763525 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.763435 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55f934f-38bb-4ea8-b721-874ab9845cdb" containerName="payload-pre-processing" Sep 07 13:29:03.766565 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.766548 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr" Sep 07 13:29:03.770097 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.770078 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jlvcf\"" Sep 07 13:29:03.770200 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.770149 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 13:29:03.786355 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.786333 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr"] Sep 07 13:29:03.860358 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.860331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10d5a54-b6d4-45ae-8d9a-90c16108dd7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr\" (UID: \"d10d5a54-b6d4-45ae-8d9a-90c16108dd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr" Sep 07 13:29:03.961451 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.961428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10d5a54-b6d4-45ae-8d9a-90c16108dd7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr\" (UID: \"d10d5a54-b6d4-45ae-8d9a-90c16108dd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr" Sep 07 13:29:03.962012 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:03.961995 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10d5a54-b6d4-45ae-8d9a-90c16108dd7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr\" (UID: \"d10d5a54-b6d4-45ae-8d9a-90c16108dd7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr" Sep 07 13:29:04.076816 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:04.076741 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr" Sep 07 13:29:04.357902 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:04.357871 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr"] Sep 07 13:29:04.359224 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:29:04.359200 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10d5a54_b6d4_45ae_8d9a_90c16108dd7b.slice/crio-b9abef08072b86cc46d663bee169fb5e11e6705e00f11d88b9b89a597157ee14 WatchSource:0}: Error finding container b9abef08072b86cc46d663bee169fb5e11e6705e00f11d88b9b89a597157ee14: Status 404 returned error can't find the container with id b9abef08072b86cc46d663bee169fb5e11e6705e00f11d88b9b89a597157ee14 Sep 07 13:29:04.509180 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:04.509146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr" event={"ID":"d10d5a54-b6d4-45ae-8d9a-90c16108dd7b","Type":"ContainerStarted","Data":"b9abef08072b86cc46d663bee169fb5e11e6705e00f11d88b9b89a597157ee14"} Sep 07 13:29:05.515760 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:05.515730 2565 generic.go:356] "Generic (PLEG): container finished" podID="d10d5a54-b6d4-45ae-8d9a-90c16108dd7b" containerID="d1ae2fea57028f59c3a7fae8c0d322a833d10e27aad5d5e93f3c7c8f710d6a88" exitCode=0 Sep 07 13:29:05.516129 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:05.515808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr" event={"ID":"d10d5a54-b6d4-45ae-8d9a-90c16108dd7b","Type":"ContainerDied","Data":"d1ae2fea57028f59c3a7fae8c0d322a833d10e27aad5d5e93f3c7c8f710d6a88"} Sep 07 13:29:06.650067 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:06.650037 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr" Sep 07 13:29:06.684225 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:06.684203 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10d5a54-b6d4-45ae-8d9a-90c16108dd7b-maas-api-service-ca\") pod \"d10d5a54-b6d4-45ae-8d9a-90c16108dd7b\" (UID: \"d10d5a54-b6d4-45ae-8d9a-90c16108dd7b\") " Sep 07 13:29:06.684471 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:06.684450 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10d5a54-b6d4-45ae-8d9a-90c16108dd7b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d10d5a54-b6d4-45ae-8d9a-90c16108dd7b" (UID: "d10d5a54-b6d4-45ae-8d9a-90c16108dd7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:29:06.785240 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:06.785175 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10d5a54-b6d4-45ae-8d9a-90c16108dd7b-maas-api-service-ca\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:07.527135 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:07.527095 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr" event={"ID":"d10d5a54-b6d4-45ae-8d9a-90c16108dd7b","Type":"ContainerDied","Data":"b9abef08072b86cc46d663bee169fb5e11e6705e00f11d88b9b89a597157ee14"} Sep 07 13:29:07.527135 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:07.527125 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr" Sep 07 13:29:07.527339 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:07.527185 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9abef08072b86cc46d663bee169fb5e11e6705e00f11d88b9b89a597157ee14" Sep 07 13:29:14.607609 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:14.607562 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq"] Sep 07 13:29:14.607998 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:14.607837 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" podUID="334fe76b-ad10-4ff5-b64a-da7ba597b1b4" containerName="payload-processing" containerID="cri-o://7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5" gracePeriod=30 Sep 07 13:29:14.864659 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:14.864512 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:29:14.866929 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:14.866904 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr"] Sep 07 13:29:14.870789 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:14.870769 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8dc72a-ngqsr"] Sep 07 13:29:14.957246 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:14.957218 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-924xn\" (UniqueName: \"kubernetes.io/projected/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-kube-api-access-924xn\") pod \"334fe76b-ad10-4ff5-b64a-da7ba597b1b4\" (UID: \"334fe76b-ad10-4ff5-b64a-da7ba597b1b4\") " Sep 07 13:29:14.957410 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:14.957321 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-plugins-config-volume\") pod \"334fe76b-ad10-4ff5-b64a-da7ba597b1b4\" (UID: \"334fe76b-ad10-4ff5-b64a-da7ba597b1b4\") " Sep 07 13:29:14.957741 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:14.957713 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "334fe76b-ad10-4ff5-b64a-da7ba597b1b4" (UID: "334fe76b-ad10-4ff5-b64a-da7ba597b1b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:29:14.959402 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:14.959380 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-kube-api-access-924xn" (OuterVolumeSpecName: "kube-api-access-924xn") pod "334fe76b-ad10-4ff5-b64a-da7ba597b1b4" (UID: "334fe76b-ad10-4ff5-b64a-da7ba597b1b4"). InnerVolumeSpecName "kube-api-access-924xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:29:15.058136 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:15.058107 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-924xn\" (UniqueName: \"kubernetes.io/projected/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-kube-api-access-924xn\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:15.058136 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:15.058134 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334fe76b-ad10-4ff5-b64a-da7ba597b1b4-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:15.569853 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:15.569815 2565 generic.go:356] "Generic (PLEG): container finished" podID="334fe76b-ad10-4ff5-b64a-da7ba597b1b4" containerID="7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5" exitCode=0 Sep 07 13:29:15.569853 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:15.569854 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" Sep 07 13:29:15.570105 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:15.569872 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" event={"ID":"334fe76b-ad10-4ff5-b64a-da7ba597b1b4","Type":"ContainerDied","Data":"7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5"} Sep 07 13:29:15.570105 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:15.569919 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq" event={"ID":"334fe76b-ad10-4ff5-b64a-da7ba597b1b4","Type":"ContainerDied","Data":"7bb0440f1d3a72b733fb6f93142f6c05f03e9d7345cb451498c2757077d6decf"} Sep 07 13:29:15.570105 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:15.569942 2565 scope.go:117] "RemoveContainer" containerID="7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5" Sep 07 13:29:15.579162 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:15.579144 2565 scope.go:117] "RemoveContainer" containerID="7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5" Sep 07 13:29:15.579413 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:29:15.579395 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5\": container with ID starting with 7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5 not found: ID does not exist" containerID="7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5" Sep 07 13:29:15.579463 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:15.579423 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5"} err="failed to get container status \"7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5\": rpc error: code = NotFound desc = could not find container \"7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5\": container with ID starting with 7a237fe7e5e5ddedf386f4ea9814d5d0a9f4e8400dc8b6e5da128cfa2bf70cd5 not found: ID does not exist" Sep 07 13:29:15.595927 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:15.595904 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq"] Sep 07 13:29:15.599897 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:15.599877 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8dc72a-679475b96f-96qpq"] Sep 07 13:29:16.826962 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:16.826928 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="334fe76b-ad10-4ff5-b64a-da7ba597b1b4" path="/var/lib/kubelet/pods/334fe76b-ad10-4ff5-b64a-da7ba597b1b4/volumes" Sep 07 13:29:16.827327 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:16.827262 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d10d5a54-b6d4-45ae-8d9a-90c16108dd7b" path="/var/lib/kubelet/pods/d10d5a54-b6d4-45ae-8d9a-90c16108dd7b/volumes" Sep 07 13:29:33.027267 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.027185 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk"] Sep 07 13:29:33.027665 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.027445 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" podUID="4b1f7bc1-96a5-4e91-836c-913ffcb2cba1" containerName="payload-processing" containerID="cri-o://e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0" gracePeriod=30 Sep 07 13:29:33.035188 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.035162 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp"] Sep 07 13:29:33.035366 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.035348 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" podUID="e02d0975-5818-4498-a1f0-3ddb5d681183" containerName="payload-pre-processing" containerID="cri-o://7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743" gracePeriod=30 Sep 07 13:29:33.310840 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.310819 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:29:33.313549 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.313533 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:29:33.443756 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.443725 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-plugins-config-volume\") pod \"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1\" (UID: \"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1\") " Sep 07 13:29:33.443923 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.443769 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02d0975-5818-4498-a1f0-3ddb5d681183-plugins-config-volume\") pod \"e02d0975-5818-4498-a1f0-3ddb5d681183\" (UID: \"e02d0975-5818-4498-a1f0-3ddb5d681183\") " Sep 07 13:29:33.443923 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.443792 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hns8\" (UniqueName: \"kubernetes.io/projected/e02d0975-5818-4498-a1f0-3ddb5d681183-kube-api-access-6hns8\") pod \"e02d0975-5818-4498-a1f0-3ddb5d681183\" (UID: \"e02d0975-5818-4498-a1f0-3ddb5d681183\") " Sep 07 13:29:33.443923 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.443823 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgwdm\" (UniqueName: \"kubernetes.io/projected/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-kube-api-access-lgwdm\") pod \"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1\" (UID: \"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1\") " Sep 07 13:29:33.444116 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.444094 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e02d0975-5818-4498-a1f0-3ddb5d681183-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e02d0975-5818-4498-a1f0-3ddb5d681183" (UID: "e02d0975-5818-4498-a1f0-3ddb5d681183"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:29:33.444185 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.444148 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b1f7bc1-96a5-4e91-836c-913ffcb2cba1" (UID: "4b1f7bc1-96a5-4e91-836c-913ffcb2cba1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:29:33.445860 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.445835 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02d0975-5818-4498-a1f0-3ddb5d681183-kube-api-access-6hns8" (OuterVolumeSpecName: "kube-api-access-6hns8") pod "e02d0975-5818-4498-a1f0-3ddb5d681183" (UID: "e02d0975-5818-4498-a1f0-3ddb5d681183"). InnerVolumeSpecName "kube-api-access-6hns8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:29:33.445956 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.445889 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-kube-api-access-lgwdm" (OuterVolumeSpecName: "kube-api-access-lgwdm") pod "4b1f7bc1-96a5-4e91-836c-913ffcb2cba1" (UID: "4b1f7bc1-96a5-4e91-836c-913ffcb2cba1"). InnerVolumeSpecName "kube-api-access-lgwdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:29:33.545364 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.545297 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:33.545364 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.545321 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e02d0975-5818-4498-a1f0-3ddb5d681183-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:33.545364 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.545330 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hns8\" (UniqueName: \"kubernetes.io/projected/e02d0975-5818-4498-a1f0-3ddb5d681183-kube-api-access-6hns8\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:33.545364 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.545339 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgwdm\" (UniqueName: \"kubernetes.io/projected/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1-kube-api-access-lgwdm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:33.675173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.675142 2565 generic.go:356] "Generic (PLEG): container finished" podID="4b1f7bc1-96a5-4e91-836c-913ffcb2cba1" containerID="e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0" exitCode=0 Sep 07 13:29:33.675317 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.675184 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" Sep 07 13:29:33.675317 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.675224 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" event={"ID":"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1","Type":"ContainerDied","Data":"e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0"} Sep 07 13:29:33.675317 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.675269 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk" event={"ID":"4b1f7bc1-96a5-4e91-836c-913ffcb2cba1","Type":"ContainerDied","Data":"12e7112b2effffecbd9325d61498f52022c1637b870d5a76c143a137833bf875"} Sep 07 13:29:33.675317 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.675286 2565 scope.go:117] "RemoveContainer" containerID="e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0" Sep 07 13:29:33.676588 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.676568 2565 generic.go:356] "Generic (PLEG): container finished" podID="e02d0975-5818-4498-a1f0-3ddb5d681183" containerID="7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743" exitCode=0 Sep 07 13:29:33.676698 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.676612 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" event={"ID":"e02d0975-5818-4498-a1f0-3ddb5d681183","Type":"ContainerDied","Data":"7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743"} Sep 07 13:29:33.676698 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.676629 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" Sep 07 13:29:33.676698 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.676657 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" event={"ID":"e02d0975-5818-4498-a1f0-3ddb5d681183","Type":"ContainerDied","Data":"fdc02035805d10b96d739ac392d4158ea074c74a7a3b2821ac34c48d69d3198d"} Sep 07 13:29:33.686292 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.686266 2565 scope.go:117] "RemoveContainer" containerID="e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0" Sep 07 13:29:33.686716 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:29:33.686685 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0\": container with ID starting with e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0 not found: ID does not exist" containerID="e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0" Sep 07 13:29:33.686833 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.686724 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0"} err="failed to get container status \"e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0\": rpc error: code = NotFound desc = could not find container \"e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0\": container with ID starting with e39afa9070d44b95467e5ceae1e156b3585ba04bb6e878b7f0e4591803cff8f0 not found: ID does not exist" Sep 07 13:29:33.686833 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.686747 2565 scope.go:117] "RemoveContainer" containerID="7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743" Sep 07 13:29:33.696978 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.696626 2565 scope.go:117] "RemoveContainer" containerID="7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743" Sep 07 13:29:33.696978 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:29:33.696886 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743\": container with ID starting with 7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743 not found: ID does not exist" containerID="7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743" Sep 07 13:29:33.696978 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.696909 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743"} err="failed to get container status \"7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743\": rpc error: code = NotFound desc = could not find container \"7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743\": container with ID starting with 7ffe933ef494687485f9c919f48b401938f5e4fee900c6445a469708ff43b743 not found: ID does not exist" Sep 07 13:29:33.712584 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.712559 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp"] Sep 07 13:29:33.716573 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.716532 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp"] Sep 07 13:29:33.730024 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.730003 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk"] Sep 07 13:29:33.732763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:33.732744 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5910bb-dfc8dd757-tt8wk"] Sep 07 13:29:34.254722 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:34.254680 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5910bb-6fd644f7fd-wm4lp" podUID="e02d0975-5818-4498-a1f0-3ddb5d681183" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.59:9004: i/o timeout" Sep 07 13:29:34.825930 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:34.825897 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b1f7bc1-96a5-4e91-836c-913ffcb2cba1" path="/var/lib/kubelet/pods/4b1f7bc1-96a5-4e91-836c-913ffcb2cba1/volumes" Sep 07 13:29:34.826249 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:34.826236 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e02d0975-5818-4498-a1f0-3ddb5d681183" path="/var/lib/kubelet/pods/e02d0975-5818-4498-a1f0-3ddb5d681183/volumes" Sep 07 13:29:41.269396 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.269359 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x"] Sep 07 13:29:41.269958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.269939 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1f7bc1-96a5-4e91-836c-913ffcb2cba1" containerName="payload-processing" Sep 07 13:29:41.270003 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.269962 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1f7bc1-96a5-4e91-836c-913ffcb2cba1" containerName="payload-processing" Sep 07 13:29:41.270003 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.269983 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334fe76b-ad10-4ff5-b64a-da7ba597b1b4" containerName="payload-processing" Sep 07 13:29:41.270003 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.269993 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="334fe76b-ad10-4ff5-b64a-da7ba597b1b4" containerName="payload-processing" Sep 07 13:29:41.270104 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.270015 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d10d5a54-b6d4-45ae-8d9a-90c16108dd7b" containerName="revoke-keys" Sep 07 13:29:41.270104 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.270024 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d10d5a54-b6d4-45ae-8d9a-90c16108dd7b" containerName="revoke-keys" Sep 07 13:29:41.270104 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.270042 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e02d0975-5818-4498-a1f0-3ddb5d681183" containerName="payload-pre-processing" Sep 07 13:29:41.270104 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.270051 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e02d0975-5818-4498-a1f0-3ddb5d681183" containerName="payload-pre-processing" Sep 07 13:29:41.270232 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.270141 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1f7bc1-96a5-4e91-836c-913ffcb2cba1" containerName="payload-processing" Sep 07 13:29:41.270232 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.270155 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d10d5a54-b6d4-45ae-8d9a-90c16108dd7b" containerName="revoke-keys" Sep 07 13:29:41.270232 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.270168 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="334fe76b-ad10-4ff5-b64a-da7ba597b1b4" containerName="payload-processing" Sep 07 13:29:41.270232 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.270179 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e02d0975-5818-4498-a1f0-3ddb5d681183" containerName="payload-pre-processing" Sep 07 13:29:41.274898 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.274877 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x" Sep 07 13:29:41.279008 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.278988 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 13:29:41.279126 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.278992 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jlvcf\"" Sep 07 13:29:41.297920 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.297895 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x"] Sep 07 13:29:41.312775 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.312747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a51c50-5b0c-4207-a4af-892cd6d6318b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x\" (UID: \"17a51c50-5b0c-4207-a4af-892cd6d6318b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x" Sep 07 13:29:41.413474 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.413444 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a51c50-5b0c-4207-a4af-892cd6d6318b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x\" (UID: \"17a51c50-5b0c-4207-a4af-892cd6d6318b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x" Sep 07 13:29:41.414056 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.414036 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a51c50-5b0c-4207-a4af-892cd6d6318b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x\" (UID: \"17a51c50-5b0c-4207-a4af-892cd6d6318b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x" Sep 07 13:29:41.585257 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.585174 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x" Sep 07 13:29:41.872453 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:41.872382 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x"] Sep 07 13:29:41.876070 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:29:41.876043 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a51c50_5b0c_4207_a4af_892cd6d6318b.slice/crio-909061672ddfc45dc927683372c2de9a4121fb4d1c0cf2759c7f33a065e83017 WatchSource:0}: Error finding container 909061672ddfc45dc927683372c2de9a4121fb4d1c0cf2759c7f33a065e83017: Status 404 returned error can't find the container with id 909061672ddfc45dc927683372c2de9a4121fb4d1c0cf2759c7f33a065e83017 Sep 07 13:29:42.726468 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:42.726430 2565 generic.go:356] "Generic (PLEG): container finished" podID="17a51c50-5b0c-4207-a4af-892cd6d6318b" containerID="a602f44f20dfe9c223184f32a10d76da31b31ce546addcf90d5d462a9bd7ad5c" exitCode=0 Sep 07 13:29:42.726865 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:42.726475 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x" event={"ID":"17a51c50-5b0c-4207-a4af-892cd6d6318b","Type":"ContainerDied","Data":"a602f44f20dfe9c223184f32a10d76da31b31ce546addcf90d5d462a9bd7ad5c"} Sep 07 13:29:42.726865 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:42.726530 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x" event={"ID":"17a51c50-5b0c-4207-a4af-892cd6d6318b","Type":"ContainerStarted","Data":"909061672ddfc45dc927683372c2de9a4121fb4d1c0cf2759c7f33a065e83017"} Sep 07 13:29:43.864506 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:43.864483 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x" Sep 07 13:29:43.934880 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:43.934851 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a51c50-5b0c-4207-a4af-892cd6d6318b-maas-api-service-ca\") pod \"17a51c50-5b0c-4207-a4af-892cd6d6318b\" (UID: \"17a51c50-5b0c-4207-a4af-892cd6d6318b\") " Sep 07 13:29:43.935215 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:43.935190 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a51c50-5b0c-4207-a4af-892cd6d6318b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17a51c50-5b0c-4207-a4af-892cd6d6318b" (UID: "17a51c50-5b0c-4207-a4af-892cd6d6318b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:29:44.036238 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:44.036166 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17a51c50-5b0c-4207-a4af-892cd6d6318b-maas-api-service-ca\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:44.737571 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:44.737538 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x" Sep 07 13:29:44.737772 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:44.737567 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x" event={"ID":"17a51c50-5b0c-4207-a4af-892cd6d6318b","Type":"ContainerDied","Data":"909061672ddfc45dc927683372c2de9a4121fb4d1c0cf2759c7f33a065e83017"} Sep 07 13:29:44.737772 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:44.737624 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="909061672ddfc45dc927683372c2de9a4121fb4d1c0cf2759c7f33a065e83017" Sep 07 13:29:52.026855 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.026822 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl"] Sep 07 13:29:52.027421 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.027043 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" podUID="acbf37ed-8edf-4104-8080-d5748bb9dacc" containerName="payload-processing" containerID="cri-o://2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d" gracePeriod=30 Sep 07 13:29:52.041384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.041356 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th"] Sep 07 13:29:52.041572 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.041538 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" podUID="1737bdfa-7386-4c9b-92fe-757724bcf4b6" containerName="payload-pre-processing" containerID="cri-o://412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d" gracePeriod=30 Sep 07 13:29:52.253213 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.252821 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x"] Sep 07 13:29:52.255256 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.255225 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c7c371-8qx7x"] Sep 07 13:29:52.312817 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.312792 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:29:52.315705 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.315685 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:29:52.404763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.404736 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbf37ed-8edf-4104-8080-d5748bb9dacc-plugins-config-volume\") pod \"acbf37ed-8edf-4104-8080-d5748bb9dacc\" (UID: \"acbf37ed-8edf-4104-8080-d5748bb9dacc\") " Sep 07 13:29:52.404943 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.404772 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1737bdfa-7386-4c9b-92fe-757724bcf4b6-plugins-config-volume\") pod \"1737bdfa-7386-4c9b-92fe-757724bcf4b6\" (UID: \"1737bdfa-7386-4c9b-92fe-757724bcf4b6\") " Sep 07 13:29:52.404943 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.404849 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlf4h\" (UniqueName: \"kubernetes.io/projected/1737bdfa-7386-4c9b-92fe-757724bcf4b6-kube-api-access-nlf4h\") pod \"1737bdfa-7386-4c9b-92fe-757724bcf4b6\" (UID: \"1737bdfa-7386-4c9b-92fe-757724bcf4b6\") " Sep 07 13:29:52.404943 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.404877 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n64rj\" (UniqueName: \"kubernetes.io/projected/acbf37ed-8edf-4104-8080-d5748bb9dacc-kube-api-access-n64rj\") pod \"acbf37ed-8edf-4104-8080-d5748bb9dacc\" (UID: \"acbf37ed-8edf-4104-8080-d5748bb9dacc\") " Sep 07 13:29:52.405225 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.405191 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbf37ed-8edf-4104-8080-d5748bb9dacc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acbf37ed-8edf-4104-8080-d5748bb9dacc" (UID: "acbf37ed-8edf-4104-8080-d5748bb9dacc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:29:52.405289 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.405216 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1737bdfa-7386-4c9b-92fe-757724bcf4b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1737bdfa-7386-4c9b-92fe-757724bcf4b6" (UID: "1737bdfa-7386-4c9b-92fe-757724bcf4b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 13:29:52.406984 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.406958 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1737bdfa-7386-4c9b-92fe-757724bcf4b6-kube-api-access-nlf4h" (OuterVolumeSpecName: "kube-api-access-nlf4h") pod "1737bdfa-7386-4c9b-92fe-757724bcf4b6" (UID: "1737bdfa-7386-4c9b-92fe-757724bcf4b6"). InnerVolumeSpecName "kube-api-access-nlf4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:29:52.407081 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.406984 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbf37ed-8edf-4104-8080-d5748bb9dacc-kube-api-access-n64rj" (OuterVolumeSpecName: "kube-api-access-n64rj") pod "acbf37ed-8edf-4104-8080-d5748bb9dacc" (UID: "acbf37ed-8edf-4104-8080-d5748bb9dacc"). InnerVolumeSpecName "kube-api-access-n64rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:29:52.506172 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.506146 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlf4h\" (UniqueName: \"kubernetes.io/projected/1737bdfa-7386-4c9b-92fe-757724bcf4b6-kube-api-access-nlf4h\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:52.506172 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.506169 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n64rj\" (UniqueName: \"kubernetes.io/projected/acbf37ed-8edf-4104-8080-d5748bb9dacc-kube-api-access-n64rj\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:52.506324 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.506182 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbf37ed-8edf-4104-8080-d5748bb9dacc-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:52.506324 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.506196 2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1737bdfa-7386-4c9b-92fe-757724bcf4b6-plugins-config-volume\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:29:52.777332 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.777297 2565 generic.go:356] "Generic (PLEG): container finished" podID="acbf37ed-8edf-4104-8080-d5748bb9dacc" containerID="2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d" exitCode=0 Sep 07 13:29:52.777506 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.777357 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" Sep 07 13:29:52.777506 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.777396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" event={"ID":"acbf37ed-8edf-4104-8080-d5748bb9dacc","Type":"ContainerDied","Data":"2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d"} Sep 07 13:29:52.777506 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.777469 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl" event={"ID":"acbf37ed-8edf-4104-8080-d5748bb9dacc","Type":"ContainerDied","Data":"0877ce1729eaf7fa2c1e390e2efd33f73c6da62f0d9d884ef2f0131b4ad28e60"} Sep 07 13:29:52.777506 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.777498 2565 scope.go:117] "RemoveContainer" containerID="2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d" Sep 07 13:29:52.779039 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.778962 2565 generic.go:356] "Generic (PLEG): container finished" podID="1737bdfa-7386-4c9b-92fe-757724bcf4b6" containerID="412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d" exitCode=0 Sep 07 13:29:52.779039 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.779024 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" Sep 07 13:29:52.779223 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.779077 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" event={"ID":"1737bdfa-7386-4c9b-92fe-757724bcf4b6","Type":"ContainerDied","Data":"412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d"} Sep 07 13:29:52.779223 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.779162 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th" event={"ID":"1737bdfa-7386-4c9b-92fe-757724bcf4b6","Type":"ContainerDied","Data":"6882bce6c6e408471a79029554b2a8184954042fcbc16e4a39c71e6603493e71"} Sep 07 13:29:52.788479 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.788462 2565 scope.go:117] "RemoveContainer" containerID="2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d" Sep 07 13:29:52.788805 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:29:52.788785 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d\": container with ID starting with 2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d not found: ID does not exist" containerID="2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d" Sep 07 13:29:52.788900 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.788816 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d"} err="failed to get container status \"2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d\": rpc error: code = NotFound desc = could not find container \"2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d\": container with ID starting with 2ef8d7cdff7b339186d82d8bd7ebe98a1ba19bdd34b36a78820944b1b77abd3d not found: ID does not exist" Sep 07 13:29:52.788900 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.788837 2565 scope.go:117] "RemoveContainer" containerID="412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d" Sep 07 13:29:52.796988 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.796973 2565 scope.go:117] "RemoveContainer" containerID="412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d" Sep 07 13:29:52.797246 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:29:52.797226 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d\": container with ID starting with 412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d not found: ID does not exist" containerID="412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d" Sep 07 13:29:52.797294 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.797252 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d"} err="failed to get container status \"412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d\": rpc error: code = NotFound desc = could not find container \"412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d\": container with ID starting with 412d2c1f2c91dd82491eabd82f3175198d84ffacf7db0605ef72aa39860bd84d not found: ID does not exist" Sep 07 13:29:52.806158 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.806132 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl"] Sep 07 13:29:52.813157 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.813137 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c7c371-f68dc447c-7vnbl"] Sep 07 13:29:52.825973 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.825953 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a51c50-5b0c-4207-a4af-892cd6d6318b" path="/var/lib/kubelet/pods/17a51c50-5b0c-4207-a4af-892cd6d6318b/volumes" Sep 07 13:29:52.826357 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.826343 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acbf37ed-8edf-4104-8080-d5748bb9dacc" path="/var/lib/kubelet/pods/acbf37ed-8edf-4104-8080-d5748bb9dacc/volumes" Sep 07 13:29:52.826862 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.826844 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th"] Sep 07 13:29:52.832104 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:52.832085 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c7c371-65ff6dbfdb-d55th"] Sep 07 13:29:54.826109 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:29:54.826076 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1737bdfa-7386-4c9b-92fe-757724bcf4b6" path="/var/lib/kubelet/pods/1737bdfa-7386-4c9b-92fe-757724bcf4b6/volumes" Sep 07 13:30:00.147812 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.147775 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv"] Sep 07 13:30:00.148203 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.148153 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a51c50-5b0c-4207-a4af-892cd6d6318b" containerName="revoke-keys" Sep 07 13:30:00.148203 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.148170 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a51c50-5b0c-4207-a4af-892cd6d6318b" containerName="revoke-keys" Sep 07 13:30:00.148203 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.148187 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1737bdfa-7386-4c9b-92fe-757724bcf4b6" containerName="payload-pre-processing" Sep 07 13:30:00.148203 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.148195 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1737bdfa-7386-4c9b-92fe-757724bcf4b6" containerName="payload-pre-processing" Sep 07 13:30:00.148375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.148215 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acbf37ed-8edf-4104-8080-d5748bb9dacc" containerName="payload-processing" Sep 07 13:30:00.148375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.148223 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="acbf37ed-8edf-4104-8080-d5748bb9dacc" containerName="payload-processing" Sep 07 13:30:00.148375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.148293 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a51c50-5b0c-4207-a4af-892cd6d6318b" containerName="revoke-keys" Sep 07 13:30:00.148375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.148303 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="acbf37ed-8edf-4104-8080-d5748bb9dacc" containerName="payload-processing" Sep 07 13:30:00.148375 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.148310 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1737bdfa-7386-4c9b-92fe-757724bcf4b6" containerName="payload-pre-processing" Sep 07 13:30:00.153039 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.153018 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv" Sep 07 13:30:00.155957 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.155935 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jlvcf\"" Sep 07 13:30:00.160449 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.160263 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv"] Sep 07 13:30:00.163836 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.163819 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv" Sep 07 13:30:00.436627 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.436586 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv"] Sep 07 13:30:00.438811 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:30:00.438784 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ad257c_e679_4f86_813a_4baddbc6e221.slice/crio-bd601248646fa9db0e1e13c4465367843c368e58a1376d70017041c4d87026fc WatchSource:0}: Error finding container bd601248646fa9db0e1e13c4465367843c368e58a1376d70017041c4d87026fc: Status 404 returned error can't find the container with id bd601248646fa9db0e1e13c4465367843c368e58a1376d70017041c4d87026fc Sep 07 13:30:00.827646 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.827615 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv" event={"ID":"35ad257c-e679-4f86-813a-4baddbc6e221","Type":"ContainerStarted","Data":"1ab58d013e385d98f42d27b9a069bcd675959cf0688ce828b2f57653b8373d22"} Sep 07 13:30:00.827809 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.827660 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv" event={"ID":"35ad257c-e679-4f86-813a-4baddbc6e221","Type":"ContainerStarted","Data":"bd601248646fa9db0e1e13c4465367843c368e58a1376d70017041c4d87026fc"} Sep 07 13:30:00.858555 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:00.858497 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv" podStartSLOduration=0.858481632 podStartE2EDuration="858.481632ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:30:00.853751666 +0000 UTC m=+1822.534137435" watchObservedRunningTime="2026-09-07 13:30:00.858481632 +0000 UTC m=+1822.538867402" Sep 07 13:30:01.832348 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:01.832311 2565 generic.go:356] "Generic (PLEG): container finished" podID="35ad257c-e679-4f86-813a-4baddbc6e221" containerID="1ab58d013e385d98f42d27b9a069bcd675959cf0688ce828b2f57653b8373d22" exitCode=0 Sep 07 13:30:01.832745 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:01.832398 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv" event={"ID":"35ad257c-e679-4f86-813a-4baddbc6e221","Type":"ContainerDied","Data":"1ab58d013e385d98f42d27b9a069bcd675959cf0688ce828b2f57653b8373d22"} Sep 07 13:30:02.967300 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:02.967277 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv" Sep 07 13:30:03.842312 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:03.842283 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv" Sep 07 13:30:03.842498 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:03.842324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813130-fmcrv" event={"ID":"35ad257c-e679-4f86-813a-4baddbc6e221","Type":"ContainerDied","Data":"bd601248646fa9db0e1e13c4465367843c368e58a1376d70017041c4d87026fc"} Sep 07 13:30:03.842498 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:30:03.842372 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd601248646fa9db0e1e13c4465367843c368e58a1376d70017041c4d87026fc" Sep 07 13:31:39.191346 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:31:39.191311 2565 scope.go:117] "RemoveContainer" containerID="842943a34188c3b4d9b8fed109d4428372462db1820defd0f7732f260ea501ec" Sep 07 13:32:39.210280 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:32:39.210192 2565 scope.go:117] "RemoveContainer" containerID="12ff6496b0ff6b8b7779be7ec91334d7e9c5ca10a2c7b6bc0d66ae82ff229eff" Sep 07 13:32:39.218269 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:32:39.218245 2565 scope.go:117] "RemoveContainer" containerID="5595f6b08108951735d2c8ef4d2e1e60736896535ca2adb611442d7976342eb4" Sep 07 13:32:39.225761 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:32:39.225744 2565 scope.go:117] "RemoveContainer" containerID="0b71826d4fa8c2fe1b86c36f726e2dd27c579b5006a27ab465c7873555f4e16e" Sep 07 13:33:42.596548 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:33:42.596501 2565 controller.go:195] "Failed to update lease" err="Put \"https://af06a7420d3144cd9ba9cdb03ad9ba9e-b2b361fbb8d6f446.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-216.ec2.internal?timeout=10s\": context deadline exceeded" Sep 07 13:33:51.776544 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:33:51.776495 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-07T13:33:41Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-07T13:33:41Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-07T13:33:41Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-07T13:33:41Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-142-216.ec2.internal\": Patch \"https://af06a7420d3144cd9ba9cdb03ad9ba9e-b2b361fbb8d6f446.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-142-216.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 07 13:33:52.596905 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:33:52.596864 2565 controller.go:195] "Failed to update lease" err="Put \"https://af06a7420d3144cd9ba9cdb03ad9ba9e-b2b361fbb8d6f446.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-216.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 07 13:34:01.777570 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:34:01.777471 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-216.ec2.internal\": Get \"https://af06a7420d3144cd9ba9cdb03ad9ba9e-b2b361fbb8d6f446.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-142-216.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 07 13:34:02.598118 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:34:02.598076 2565 controller.go:195] "Failed to update lease" err="Put \"https://af06a7420d3144cd9ba9cdb03ad9ba9e-b2b361fbb8d6f446.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-216.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 07 13:34:11.778656 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:34:11.778619 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-216.ec2.internal\": Get \"https://af06a7420d3144cd9ba9cdb03ad9ba9e-b2b361fbb8d6f446.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-142-216.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 07 13:34:12.599220 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:34:12.599180 2565 controller.go:195] "Failed to update lease" err="Put \"https://af06a7420d3144cd9ba9cdb03ad9ba9e-b2b361fbb8d6f446.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-216.ec2.internal?timeout=10s\": context deadline exceeded" Sep 07 13:34:13.815207 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815165 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815207 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815165 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gncsz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815207 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815197 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8d5qt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815207 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815171 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815233 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815242 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kcxq5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815254 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815277 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815293 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815308 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815320 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815313 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815327 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815356 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h7547\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815373 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815398 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nszhg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815407 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815416 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815437 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n5q6z\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815438 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815448 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.815936 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815459 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815467 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dn6gs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815478 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815491 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815513 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815526 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cxmkd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815539 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815556 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815560 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815621 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815634 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p9c22\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815643 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815656 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815651 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815702 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815702 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815710 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815729 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.816883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815752 2565 reflector.go:571] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815751 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815768 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815781 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815791 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815802 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815819 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815830 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-45cgz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815832 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815863 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-52dkl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815863 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815861 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815890 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815912 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815914 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815957 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815972 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5v9xw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815993 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.817569 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815998 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.815999 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816018 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816024 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816044 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816047 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816055 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816076 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816083 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816089 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816098 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k4vwh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816114 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816114 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816157 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816174 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816180 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2dhv4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816192 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rqgx2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816194 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818173 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816218 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dn6dpcuqkqds\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816225 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lbqzv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816231 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rdx7q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816237 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dthxs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816244 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816231 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816250 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gp6md\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816252 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-czmdf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816253 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816277 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816281 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816292 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816301 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pp6wd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816312 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816320 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816332 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816338 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816360 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.818763 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816364 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816364 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816374 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816385 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816382 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6trkj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816401 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816403 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816411 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-l7ztq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816416 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816424 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vvqfs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816429 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816426 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816431 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816443 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-oqjlac5mgcsa\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816450 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816456 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816463 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816473 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819384 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816470 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s2pjd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816498 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816512 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816517 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"default-dockercfg-df9ht\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816526 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:34:13.816530 2565 controller.go:195] "Failed to update lease" err="Put \"https://af06a7420d3144cd9ba9cdb03ad9ba9e-b2b361fbb8d6f446.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-142-216.ec2.internal?timeout=10s\": http2: client connection lost" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816542 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816543 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816555 2565 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816499 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816566 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816571 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816575 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"default-dockercfg-lmrpz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816541 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816584 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kgmtr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816614 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816575 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816618 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z6v8j\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816617 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.819958 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816628 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816631 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816541 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7iqmuc3l4cbg8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816649 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816655 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816645 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816674 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816679 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-dockercfg-td2kc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816685 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816684 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2jq42\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816704 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816706 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bjsqg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816687 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816717 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9l2d6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816716 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816721 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816737 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816744 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.820626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816743 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.821182 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816747 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.821182 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816756 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.821182 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816758 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:13.821182 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:13.816761 2565 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 07 13:34:14.628702 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.628668 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 07 13:34:14.628702 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.628669 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 07 13:34:14.629916 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.629899 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dn6gs\"" Sep 07 13:34:14.630629 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.630611 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dn6dpcuqkqds\"" Sep 07 13:34:14.641424 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.641397 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 07 13:34:14.645353 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.645336 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 07 13:34:14.647940 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.647910 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 07 13:34:14.648004 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.647995 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bjsqg\"" Sep 07 13:34:14.653586 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.653568 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6trkj\"" Sep 07 13:34:14.661875 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.661859 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 07 13:34:14.674495 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.674469 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2dhv4\"" Sep 07 13:34:14.674666 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.674648 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-oqjlac5mgcsa\"" Sep 07 13:34:14.680065 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.680045 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vvqfs\"" Sep 07 13:34:14.690805 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.690785 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 07 13:34:14.690892 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.690835 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 07 13:34:14.701510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.701488 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 07 13:34:14.711771 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.711742 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 07 13:34:14.718283 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.718266 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 07 13:34:14.718379 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.718332 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 07 13:34:14.718510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.718497 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 13:34:14.718510 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.718504 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 07 13:34:14.726300 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.726283 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 07 13:34:14.728052 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.728038 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 07 13:34:14.741897 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.741879 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 07 13:34:14.748857 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.748840 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 07 13:34:14.749655 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.749642 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 07 13:34:14.758904 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.758889 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 07 13:34:14.759019 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.759001 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pp6wd\"" Sep 07 13:34:14.759493 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.759478 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 07 13:34:14.760528 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.760513 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h7547\"" Sep 07 13:34:14.764878 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.764865 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 07 13:34:14.765813 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.765799 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 07 13:34:14.768919 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.768904 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 07 13:34:14.769746 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.769733 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 07 13:34:14.774955 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.774940 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 07 13:34:14.778499 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.778481 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 07 13:34:14.792305 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.792285 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 07 13:34:14.797289 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.797273 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gp6md\"" Sep 07 13:34:14.798162 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.798149 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 07 13:34:14.803887 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.803873 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2jq42\"" Sep 07 13:34:14.813331 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.813310 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lmrpz\"" Sep 07 13:34:14.834298 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.834277 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 07 13:34:14.834572 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.834308 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 07 13:34:14.834572 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.834371 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kgmtr\"" Sep 07 13:34:14.850634 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.850614 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 07 13:34:14.851718 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.851705 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 07 13:34:14.864398 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.864379 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 07 13:34:14.868631 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.868615 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 07 13:34:14.874634 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.874619 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 07 13:34:14.875442 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.875428 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 07 13:34:14.880181 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.880140 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-45cgz\"" Sep 07 13:34:14.883781 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.883767 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 07 13:34:14.887684 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.887670 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n5q6z\"" Sep 07 13:34:14.901022 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.901006 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 07 13:34:14.901186 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.901173 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7iqmuc3l4cbg8\"" Sep 07 13:34:14.911138 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.911117 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 07 13:34:14.911238 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.911224 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-df9ht\"" Sep 07 13:34:14.922226 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.922204 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 07 13:34:14.923878 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.923862 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p9c22\"" Sep 07 13:34:14.925992 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.925975 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 07 13:34:14.926093 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.926019 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 07 13:34:14.947020 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.946999 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 07 13:34:14.958931 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.958915 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 07 13:34:14.959672 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.959650 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 07 13:34:14.959734 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.959692 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 07 13:34:14.964182 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.964165 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 07 13:34:14.965458 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.965443 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 07 13:34:14.966962 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.966945 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 07 13:34:14.972899 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.972884 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k4vwh\"" Sep 07 13:34:14.981142 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.981124 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lbqzv\"" Sep 07 13:34:14.987624 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.987605 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 07 13:34:14.993546 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.993524 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-52dkl\"" Sep 07 13:34:14.996694 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.996676 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 07 13:34:14.996790 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:14.996722 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 07 13:34:15.010648 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.010632 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 13:34:15.023194 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.023179 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-czmdf\"" Sep 07 13:34:15.026512 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.026491 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 07 13:34:15.041316 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.041296 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 07 13:34:15.051713 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.051694 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kcxq5\"" Sep 07 13:34:15.059374 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.059359 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 07 13:34:15.060456 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.060441 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 07 13:34:15.065466 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.065452 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 07 13:34:15.077408 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.077389 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 07 13:34:15.100947 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.100927 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 07 13:34:15.101970 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.101955 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8d5qt\"" Sep 07 13:34:15.114847 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.114820 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 07 13:34:15.115056 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.115041 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-td2kc\"" Sep 07 13:34:15.124518 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.124499 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 07 13:34:15.128267 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.128250 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 07 13:34:15.129395 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.129381 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rdx7q\"" Sep 07 13:34:15.131168 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.131121 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 07 13:34:15.134871 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.134854 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 07 13:34:15.143372 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.143351 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 07 13:34:15.147073 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.147055 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 07 13:34:15.147864 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.147850 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cxmkd\"" Sep 07 13:34:15.151333 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.151319 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 13:34:15.154736 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.154720 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 07 13:34:15.156764 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.156748 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 07 13:34:15.163345 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.163328 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 07 13:34:15.167448 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.167430 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 07 13:34:15.176105 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.176084 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 07 13:34:15.178909 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.178894 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 07 13:34:15.179111 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.179098 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s2pjd\"" Sep 07 13:34:15.191479 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.191460 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gncsz\"" Sep 07 13:34:15.193482 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.193466 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 07 13:34:15.195179 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.195162 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 07 13:34:15.199685 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.199670 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 07 13:34:15.206812 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.206797 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 07 13:34:15.212627 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.212611 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 13:34:15.212843 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.212828 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 07 13:34:15.217391 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.217374 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nszhg\"" Sep 07 13:34:15.228624 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.228588 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 07 13:34:15.233704 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.233688 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 07 13:34:15.234516 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.234501 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 07 13:34:15.236534 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.236520 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dthxs\"" Sep 07 13:34:15.241233 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.241216 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 07 13:34:15.247231 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.247217 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 07 13:34:15.252242 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.252227 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z6v8j\"" Sep 07 13:34:15.255380 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.255365 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 07 13:34:15.264137 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.264119 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 07 13:34:15.266259 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.266242 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 07 13:34:15.278793 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.278775 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5v9xw\"" Sep 07 13:34:15.279837 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.279820 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 07 13:34:15.280851 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.280835 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 07 13:34:15.285239 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.285223 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-l7ztq\"" Sep 07 13:34:15.286343 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.286324 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 07 13:34:15.288882 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.288866 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 13:34:15.291898 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.291878 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 07 13:34:15.294855 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.294836 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 07 13:34:15.305606 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.305579 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 07 13:34:15.319187 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.319169 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 07 13:34:15.333807 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.333790 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 07 13:34:15.345786 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.345768 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 07 13:34:15.352158 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.352141 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 07 13:34:15.352643 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.352628 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 07 13:34:15.353294 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.353278 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 07 13:34:15.373478 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.373457 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 07 13:34:15.388626 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.388553 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9l2d6\"" Sep 07 13:34:15.406872 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.406853 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 07 13:34:15.426776 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.426757 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 07 13:34:15.447396 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.447375 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 07 13:34:15.467841 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.467822 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 07 13:34:15.487096 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.487078 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rqgx2\"" Sep 07 13:34:15.508170 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.508152 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 07 13:34:15.527296 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.527275 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 07 13:34:15.546727 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.546703 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 07 13:34:15.567400 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.567380 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 07 13:34:15.587586 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.587570 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 07 13:34:15.606950 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.606929 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 13:34:15.627414 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.627398 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 07 13:34:15.648312 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:15.648295 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 07 13:34:39.257846 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:39.257813 2565 scope.go:117] "RemoveContainer" containerID="68921f916be81a2de8d207624e9bc325c32b0b96346e89e8f8b0cff4078d0037" Sep 07 13:34:39.266165 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:39.266145 2565 scope.go:117] "RemoveContainer" containerID="bca0c1c38477f52574dcb8feecafbad4d72f76f81e08014986c23179cfd0626c" Sep 07 13:34:55.962783 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:55.962744 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8"] Sep 07 13:34:55.963225 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:55.963105 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ad257c-e679-4f86-813a-4baddbc6e221" containerName="cleanup" Sep 07 13:34:55.963225 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:55.963118 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ad257c-e679-4f86-813a-4baddbc6e221" containerName="cleanup" Sep 07 13:34:55.963225 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:55.963183 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="35ad257c-e679-4f86-813a-4baddbc6e221" containerName="cleanup" Sep 07 13:34:55.967622 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:55.967247 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:55.972334 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:55.972310 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bfd282e0-kserve-self-signed-certs\"" Sep 07 13:34:56.009004 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.008966 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8"] Sep 07 13:34:56.096919 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.096882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnpnd\" (UniqueName: \"kubernetes.io/projected/2a10ef0a-9019-4986-99b9-7219bd984a78-kube-api-access-cnpnd\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.096919 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.096924 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-home\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.097139 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.096986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-dshm\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.097139 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.097059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-tmp-dir\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.097139 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.097097 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a10ef0a-9019-4986-99b9-7219bd984a78-tls-certs\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.097139 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.097119 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-model-cache\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.198538 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.198502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnpnd\" (UniqueName: \"kubernetes.io/projected/2a10ef0a-9019-4986-99b9-7219bd984a78-kube-api-access-cnpnd\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.198538 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.198542 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-home\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.198789 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.198615 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-dshm\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.198789 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.198661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-tmp-dir\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.198789 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.198686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a10ef0a-9019-4986-99b9-7219bd984a78-tls-certs\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.198789 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.198714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-model-cache\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.199065 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.199040 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-home\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.199131 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.199090 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-tmp-dir\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.199131 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.199105 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-model-cache\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.200970 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.200943 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-dshm\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.201087 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.201069 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a10ef0a-9019-4986-99b9-7219bd984a78-tls-certs\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.207869 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.207845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnpnd\" (UniqueName: \"kubernetes.io/projected/2a10ef0a-9019-4986-99b9-7219bd984a78-kube-api-access-cnpnd\") pod \"e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.278225 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.278131 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:34:56.554380 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:56.554358 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8"] Sep 07 13:34:56.556720 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:34:56.556689 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a10ef0a_9019_4986_99b9_7219bd984a78.slice/crio-efa8abe24278d54af22a6f397dcc3472e6779a27348970d8aaa29552b68e72ab WatchSource:0}: Error finding container efa8abe24278d54af22a6f397dcc3472e6779a27348970d8aaa29552b68e72ab: Status 404 returned error can't find the container with id efa8abe24278d54af22a6f397dcc3472e6779a27348970d8aaa29552b68e72ab Sep 07 13:34:57.279005 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:57.278965 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" event={"ID":"2a10ef0a-9019-4986-99b9-7219bd984a78","Type":"ContainerStarted","Data":"effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18"} Sep 07 13:34:57.279402 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:34:57.279014 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" event={"ID":"2a10ef0a-9019-4986-99b9-7219bd984a78","Type":"ContainerStarted","Data":"efa8abe24278d54af22a6f397dcc3472e6779a27348970d8aaa29552b68e72ab"} Sep 07 13:35:02.923658 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:02.923588 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" podStartSLOduration=7.923568689 podStartE2EDuration="7.923568689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:34:57.310719417 +0000 UTC m=+2118.991105186" watchObservedRunningTime="2026-09-07 13:35:02.923568689 +0000 UTC m=+2124.603954458" Sep 07 13:35:02.924650 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:02.924627 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5"] Sep 07 13:35:02.928150 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:02.928131 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:02.933925 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:02.933897 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-bfd282e0-kserve-self-signed-certs\"" Sep 07 13:35:03.064742 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.064709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tmp-dir\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.064928 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.064758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8z7q\" (UniqueName: \"kubernetes.io/projected/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-kube-api-access-p8z7q\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.064928 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.064793 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-dshm\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.064928 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.064814 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-home\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.064928 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.064897 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tls-certs\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.065121 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.064956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-model-cache\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.086787 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.086759 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5"] Sep 07 13:35:03.166134 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.166100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tls-certs\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.166300 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.166163 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-model-cache\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.166300 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.166198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tmp-dir\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.166300 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.166218 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8z7q\" (UniqueName: \"kubernetes.io/projected/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-kube-api-access-p8z7q\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.166300 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.166241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-dshm\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.166300 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.166257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-home\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.166646 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.166620 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-home\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.166646 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.166631 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-model-cache\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.166769 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.166693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tmp-dir\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.168532 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.168507 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-dshm\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.168814 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.168796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tls-certs\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.179590 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.179543 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8z7q\" (UniqueName: \"kubernetes.io/projected/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-kube-api-access-p8z7q\") pod \"e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.237274 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.237241 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:03.526653 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:35:03.526619 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4ff6b22_faa5_4eb3_bf51_850e0f5f4a19.slice/crio-f180e2f68052c13ee88d4a9fd2f075ed364adb5551d308eebbcf44608d4ec465 WatchSource:0}: Error finding container f180e2f68052c13ee88d4a9fd2f075ed364adb5551d308eebbcf44608d4ec465: Status 404 returned error can't find the container with id f180e2f68052c13ee88d4a9fd2f075ed364adb5551d308eebbcf44608d4ec465 Sep 07 13:35:03.526871 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:03.526850 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5"] Sep 07 13:35:04.315388 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:04.315353 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" event={"ID":"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19","Type":"ContainerStarted","Data":"a90a700ddd74be9169cdaafc5d753d4a39f4965e5c070601040bd2b99140848a"} Sep 07 13:35:04.315388 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:04.315390 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" event={"ID":"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19","Type":"ContainerStarted","Data":"f180e2f68052c13ee88d4a9fd2f075ed364adb5551d308eebbcf44608d4ec465"} Sep 07 13:35:06.278716 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:06.278672 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:35:06.279115 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:06.278738 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:35:06.293179 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:06.293147 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:35:06.332404 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:06.332350 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" podStartSLOduration=4.332333685 podStartE2EDuration="4.332333685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:35:04.361150602 +0000 UTC m=+2126.041536372" watchObservedRunningTime="2026-09-07 13:35:06.332333685 +0000 UTC m=+2128.012719455" Sep 07 13:35:06.337378 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:06.337351 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:35:13.237535 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:13.237499 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:13.237535 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:13.237546 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:13.249960 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:13.249935 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:13.370536 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:13.370507 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:31.534201 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:31.534122 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8"] Sep 07 13:35:33.262940 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:33.262910 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5"] Sep 07 13:35:39.290383 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:39.290267 2565 scope.go:117] "RemoveContainer" containerID="d1ae2fea57028f59c3a7fae8c0d322a833d10e27aad5d5e93f3c7c8f710d6a88" Sep 07 13:35:46.553464 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.553421 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" podUID="2a10ef0a-9019-4986-99b9-7219bd984a78" containerName="main" containerID="cri-o://effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18" gracePeriod=45 Sep 07 13:35:46.797320 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.797294 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:35:46.875364 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875304 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnpnd\" (UniqueName: \"kubernetes.io/projected/2a10ef0a-9019-4986-99b9-7219bd984a78-kube-api-access-cnpnd\") pod \"2a10ef0a-9019-4986-99b9-7219bd984a78\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " Sep 07 13:35:46.875364 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875362 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-tmp-dir\") pod \"2a10ef0a-9019-4986-99b9-7219bd984a78\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " Sep 07 13:35:46.875536 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875392 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a10ef0a-9019-4986-99b9-7219bd984a78-tls-certs\") pod \"2a10ef0a-9019-4986-99b9-7219bd984a78\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " Sep 07 13:35:46.875536 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875419 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-model-cache\") pod \"2a10ef0a-9019-4986-99b9-7219bd984a78\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " Sep 07 13:35:46.875536 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875445 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-home\") pod \"2a10ef0a-9019-4986-99b9-7219bd984a78\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " Sep 07 13:35:46.875536 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875507 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-dshm\") pod \"2a10ef0a-9019-4986-99b9-7219bd984a78\" (UID: \"2a10ef0a-9019-4986-99b9-7219bd984a78\") " Sep 07 13:35:46.875779 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875697 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a10ef0a-9019-4986-99b9-7219bd984a78" (UID: "2a10ef0a-9019-4986-99b9-7219bd984a78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:35:46.875843 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875766 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a10ef0a-9019-4986-99b9-7219bd984a78" (UID: "2a10ef0a-9019-4986-99b9-7219bd984a78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:35:46.875843 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875783 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-home" (OuterVolumeSpecName: "home") pod "2a10ef0a-9019-4986-99b9-7219bd984a78" (UID: "2a10ef0a-9019-4986-99b9-7219bd984a78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:35:46.875941 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875863 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-model-cache\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:46.875941 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875881 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-home\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:46.875941 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.875895 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-tmp-dir\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:46.877529 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.877511 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a10ef0a-9019-4986-99b9-7219bd984a78-kube-api-access-cnpnd" (OuterVolumeSpecName: "kube-api-access-cnpnd") pod "2a10ef0a-9019-4986-99b9-7219bd984a78" (UID: "2a10ef0a-9019-4986-99b9-7219bd984a78"). InnerVolumeSpecName "kube-api-access-cnpnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:35:46.878022 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.877997 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a10ef0a-9019-4986-99b9-7219bd984a78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a10ef0a-9019-4986-99b9-7219bd984a78" (UID: "2a10ef0a-9019-4986-99b9-7219bd984a78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:35:46.878106 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.878029 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-dshm" (OuterVolumeSpecName: "dshm") pod "2a10ef0a-9019-4986-99b9-7219bd984a78" (UID: "2a10ef0a-9019-4986-99b9-7219bd984a78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:35:46.977222 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.977187 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a10ef0a-9019-4986-99b9-7219bd984a78-dshm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:46.977222 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.977217 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnpnd\" (UniqueName: \"kubernetes.io/projected/2a10ef0a-9019-4986-99b9-7219bd984a78-kube-api-access-cnpnd\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:46.977413 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:46.977232 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a10ef0a-9019-4986-99b9-7219bd984a78-tls-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:47.518049 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:47.518018 2565 generic.go:356] "Generic (PLEG): container finished" podID="2a10ef0a-9019-4986-99b9-7219bd984a78" containerID="effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18" exitCode=0 Sep 07 13:35:47.518221 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:47.518060 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" Sep 07 13:35:47.518221 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:47.518087 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" event={"ID":"2a10ef0a-9019-4986-99b9-7219bd984a78","Type":"ContainerDied","Data":"effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18"} Sep 07 13:35:47.518221 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:47.518117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8" event={"ID":"2a10ef0a-9019-4986-99b9-7219bd984a78","Type":"ContainerDied","Data":"efa8abe24278d54af22a6f397dcc3472e6779a27348970d8aaa29552b68e72ab"} Sep 07 13:35:47.518221 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:47.518132 2565 scope.go:117] "RemoveContainer" containerID="effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18" Sep 07 13:35:47.529813 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:47.529790 2565 scope.go:117] "RemoveContainer" containerID="effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18" Sep 07 13:35:47.530310 ip-10-0-142-216 kubenswrapper[2565]: E0907 13:35:47.530275 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18\": container with ID starting with effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18 not found: ID does not exist" containerID="effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18" Sep 07 13:35:47.530381 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:47.530321 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18"} err="failed to get container status \"effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18\": rpc error: code = NotFound desc = could not find container \"effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18\": container with ID starting with effe31be50dcc061c4c128de5002f5b40da5f54e25da5841eb2de9a179d61c18 not found: ID does not exist" Sep 07 13:35:47.544666 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:47.544641 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8"] Sep 07 13:35:47.547421 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:47.547399 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bfd282e0-kserve-d867f6f6c-88fl8"] Sep 07 13:35:48.282081 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.282046 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" podUID="c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" containerName="main" containerID="cri-o://a90a700ddd74be9169cdaafc5d753d4a39f4965e5c070601040bd2b99140848a" gracePeriod=45 Sep 07 13:35:48.524371 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.524343 2565 generic.go:356] "Generic (PLEG): container finished" podID="c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" containerID="a90a700ddd74be9169cdaafc5d753d4a39f4965e5c070601040bd2b99140848a" exitCode=0 Sep 07 13:35:48.524485 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.524410 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" event={"ID":"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19","Type":"ContainerDied","Data":"a90a700ddd74be9169cdaafc5d753d4a39f4965e5c070601040bd2b99140848a"} Sep 07 13:35:48.524485 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.524448 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" event={"ID":"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19","Type":"ContainerDied","Data":"f180e2f68052c13ee88d4a9fd2f075ed364adb5551d308eebbcf44608d4ec465"} Sep 07 13:35:48.524485 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.524461 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f180e2f68052c13ee88d4a9fd2f075ed364adb5551d308eebbcf44608d4ec465" Sep 07 13:35:48.525513 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.525496 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:48.589393 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589336 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tls-certs\") pod \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " Sep 07 13:35:48.589520 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589432 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-home\") pod \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " Sep 07 13:35:48.589520 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589474 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tmp-dir\") pod \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " Sep 07 13:35:48.589520 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589502 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-model-cache\") pod \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " Sep 07 13:35:48.589690 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589549 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8z7q\" (UniqueName: \"kubernetes.io/projected/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-kube-api-access-p8z7q\") pod \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " Sep 07 13:35:48.589690 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589620 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-dshm\") pod \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\" (UID: \"c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19\") " Sep 07 13:35:48.589799 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589772 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-home" (OuterVolumeSpecName: "home") pod "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" (UID: "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:35:48.589859 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589801 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" (UID: "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:35:48.589859 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589813 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" (UID: "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:35:48.589978 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589964 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-home\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:48.590047 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589981 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tmp-dir\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:48.590047 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.589990 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-model-cache\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:48.591534 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.591508 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" (UID: "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 13:35:48.591666 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.591640 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-dshm" (OuterVolumeSpecName: "dshm") pod "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" (UID: "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 13:35:48.591743 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.591668 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-kube-api-access-p8z7q" (OuterVolumeSpecName: "kube-api-access-p8z7q") pod "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" (UID: "c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19"). InnerVolumeSpecName "kube-api-access-p8z7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 13:35:48.691142 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.691095 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8z7q\" (UniqueName: \"kubernetes.io/projected/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-kube-api-access-p8z7q\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:48.691142 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.691123 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-dshm\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:48.691142 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.691134 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19-tls-certs\") on node \"ip-10-0-142-216.ec2.internal\" DevicePath \"\"" Sep 07 13:35:48.826212 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:48.826177 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a10ef0a-9019-4986-99b9-7219bd984a78" path="/var/lib/kubelet/pods/2a10ef0a-9019-4986-99b9-7219bd984a78/volumes" Sep 07 13:35:49.529309 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:49.529283 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5" Sep 07 13:35:49.547649 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:49.547617 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5"] Sep 07 13:35:49.553263 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:49.553238 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-bfd282e0-kserve-85995658f5-ffhc5"] Sep 07 13:35:50.826802 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:35:50.826766 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" path="/var/lib/kubelet/pods/c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19/volumes" Sep 07 13:36:10.081881 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:10.081839 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-zbw5l" podUID="96aef354-0582-4a9d-b427-4f347a152da2" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 13:36:10.082432 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:10.081976 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zbw5l" Sep 07 13:36:10.082562 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:10.082537 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-zbw5l" podUID="96aef354-0582-4a9d-b427-4f347a152da2" containerName="konnectivity-agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 13:36:22.874677 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:22.874647 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6598d66d5b-6rhbr_698d6a55-c084-40aa-8358-f0149a0fc6ad/authorino/0.log" Sep 07 13:36:29.265874 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:29.265843 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5545ff49d6-j24tl_2eae9c0e-b88a-4a38-9a5c-e58c2c744c38/copy-manifests/0.log" Sep 07 13:36:29.282341 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:29.282296 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5545ff49d6-j24tl_2eae9c0e-b88a-4a38-9a5c-e58c2c744c38/manager/0.log" Sep 07 13:36:30.125774 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:30.125743 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29813130-fmcrv_35ad257c-e679-4f86-813a-4baddbc6e221/cleanup/0.log" Sep 07 13:36:31.039650 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:31.039616 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6598d66d5b-6rhbr_698d6a55-c084-40aa-8358-f0149a0fc6ad/authorino/0.log" Sep 07 13:36:31.762925 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:31.762888 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-n42fq_a6a5e5de-a0ca-45cf-b61d-c82ade4c22f4/manager/0.log" Sep 07 13:36:32.504957 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:32.504928 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546b89bbb7-spq8c_2fc20259-3152-445f-b813-f6dc5da42ed1/kube-auth-proxy/0.log" Sep 07 13:36:32.766630 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:32.766541 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6bld5_6009be81-7b8d-44b4-8fa4-d8c3da6736cc/payload-pre-processing/0.log" Sep 07 13:36:33.606663 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:33.606632 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-bmmch_a46af2c4-393c-485d-84ef-53e01d008442/main/0.log" Sep 07 13:36:33.734342 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:33.734302 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tmwvz_9ad73588-dd84-4766-9621-4bcfea48abe4/main/0.log" Sep 07 13:36:34.115483 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:34.115451 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-2b7cg_b3e3c31e-5117-41a8-8e12-a3dcf555f2f7/main/0.log" Sep 07 13:36:39.309396 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:39.309365 2565 scope.go:117] "RemoveContainer" containerID="a602f44f20dfe9c223184f32a10d76da31b31ce546addcf90d5d462a9bd7ad5c" Sep 07 13:36:40.082991 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.082961 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zbw5l" Sep 07 13:36:40.305675 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.305639 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-scn9z/must-gather-5sdvp"] Sep 07 13:36:40.306037 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.306025 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a10ef0a-9019-4986-99b9-7219bd984a78" containerName="main" Sep 07 13:36:40.306082 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.306038 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a10ef0a-9019-4986-99b9-7219bd984a78" containerName="main" Sep 07 13:36:40.306082 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.306073 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" containerName="main" Sep 07 13:36:40.306082 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.306080 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" containerName="main" Sep 07 13:36:40.306181 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.306138 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4ff6b22-faa5-4eb3-bf51-850e0f5f4a19" containerName="main" Sep 07 13:36:40.306181 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.306150 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a10ef0a-9019-4986-99b9-7219bd984a78" containerName="main" Sep 07 13:36:40.310628 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.310589 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-scn9z/must-gather-5sdvp" Sep 07 13:36:40.313677 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.313659 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-scn9z\"/\"openshift-service-ca.crt\"" Sep 07 13:36:40.313844 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.313823 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-scn9z\"/\"kube-root-ca.crt\"" Sep 07 13:36:40.323864 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.323842 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-scn9z/must-gather-5sdvp"] Sep 07 13:36:40.458884 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.458845 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/119729e4-e0d6-413c-9526-5a09c04f76c3-must-gather-output\") pod \"must-gather-5sdvp\" (UID: \"119729e4-e0d6-413c-9526-5a09c04f76c3\") " pod="openshift-must-gather-scn9z/must-gather-5sdvp" Sep 07 13:36:40.459053 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.458902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsld4\" (UniqueName: \"kubernetes.io/projected/119729e4-e0d6-413c-9526-5a09c04f76c3-kube-api-access-wsld4\") pod \"must-gather-5sdvp\" (UID: \"119729e4-e0d6-413c-9526-5a09c04f76c3\") " pod="openshift-must-gather-scn9z/must-gather-5sdvp" Sep 07 13:36:40.559607 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.559571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsld4\" (UniqueName: \"kubernetes.io/projected/119729e4-e0d6-413c-9526-5a09c04f76c3-kube-api-access-wsld4\") pod \"must-gather-5sdvp\" (UID: \"119729e4-e0d6-413c-9526-5a09c04f76c3\") " pod="openshift-must-gather-scn9z/must-gather-5sdvp" Sep 07 13:36:40.559748 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.559667 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/119729e4-e0d6-413c-9526-5a09c04f76c3-must-gather-output\") pod \"must-gather-5sdvp\" (UID: \"119729e4-e0d6-413c-9526-5a09c04f76c3\") " pod="openshift-must-gather-scn9z/must-gather-5sdvp" Sep 07 13:36:40.559952 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.559934 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/119729e4-e0d6-413c-9526-5a09c04f76c3-must-gather-output\") pod \"must-gather-5sdvp\" (UID: \"119729e4-e0d6-413c-9526-5a09c04f76c3\") " pod="openshift-must-gather-scn9z/must-gather-5sdvp" Sep 07 13:36:40.569084 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.569055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsld4\" (UniqueName: \"kubernetes.io/projected/119729e4-e0d6-413c-9526-5a09c04f76c3-kube-api-access-wsld4\") pod \"must-gather-5sdvp\" (UID: \"119729e4-e0d6-413c-9526-5a09c04f76c3\") " pod="openshift-must-gather-scn9z/must-gather-5sdvp" Sep 07 13:36:40.620002 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.619965 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-scn9z/must-gather-5sdvp" Sep 07 13:36:40.902676 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.902652 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-scn9z/must-gather-5sdvp"] Sep 07 13:36:40.904677 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:36:40.904647 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod119729e4_e0d6_413c_9526_5a09c04f76c3.slice/crio-a3bc8dd88c9a5ce5f6449aa8b6762584ce3ddba323045dc79d6389128cc719f4 WatchSource:0}: Error finding container a3bc8dd88c9a5ce5f6449aa8b6762584ce3ddba323045dc79d6389128cc719f4: Status 404 returned error can't find the container with id a3bc8dd88c9a5ce5f6449aa8b6762584ce3ddba323045dc79d6389128cc719f4 Sep 07 13:36:40.906698 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:40.906677 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 13:36:41.778004 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:41.777978 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-scn9z/must-gather-5sdvp" event={"ID":"119729e4-e0d6-413c-9526-5a09c04f76c3","Type":"ContainerStarted","Data":"0852fbedbc21632c20d9ffb00a6a143a4366da14850a2804c5a091327dd2274f"} Sep 07 13:36:41.778377 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:41.778022 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-scn9z/must-gather-5sdvp" event={"ID":"119729e4-e0d6-413c-9526-5a09c04f76c3","Type":"ContainerStarted","Data":"a3bc8dd88c9a5ce5f6449aa8b6762584ce3ddba323045dc79d6389128cc719f4"} Sep 07 13:36:42.786530 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:42.786493 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-scn9z/must-gather-5sdvp" event={"ID":"119729e4-e0d6-413c-9526-5a09c04f76c3","Type":"ContainerStarted","Data":"64f34784c799800495d6e4aafc5a957ab7714d19732606851cb9be98556af37b"} Sep 07 13:36:42.810829 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:42.810770 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-scn9z/must-gather-5sdvp" podStartSLOduration=2.045887765 podStartE2EDuration="2.810747695s" totalImagesPullingTime="764.85993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 13:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:36:42.808865269 +0000 UTC m=+2224.489251040" watchObservedRunningTime="2026-09-07 13:36:42.810747695 +0000 UTC m=+2224.491133465" Sep 07 13:36:43.494775 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:43.494745 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7nlgb_b2143bf9-8156-489e-9d51-cdee196d4ea7/global-pull-secret-syncer/0.log" Sep 07 13:36:43.823956 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:43.823886 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zbw5l_96aef354-0582-4a9d-b427-4f347a152da2/konnectivity-agent/0.log" Sep 07 13:36:43.894299 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:43.894264 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-216.ec2.internal_1e5b89f6dd594d68f12f82f4340ea7e5/haproxy/0.log" Sep 07 13:36:48.867310 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:48.867274 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6598d66d5b-6rhbr_698d6a55-c084-40aa-8358-f0149a0fc6ad/authorino/0.log" Sep 07 13:36:49.213616 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:49.213559 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-n42fq_a6a5e5de-a0ca-45cf-b61d-c82ade4c22f4/manager/0.log" Sep 07 13:36:50.911161 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:50.911121 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-q8dpn_1ab8a00e-df57-45f1-b68c-2198e83caf41/cluster-monitoring-operator/0.log" Sep 07 13:36:50.962344 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:50.962315 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-64j7z_06b6ce26-951d-44da-ae1d-4b20971557fd/kube-state-metrics/0.log" Sep 07 13:36:50.997876 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:50.997833 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-64j7z_06b6ce26-951d-44da-ae1d-4b20971557fd/kube-rbac-proxy-main/0.log" Sep 07 13:36:51.043737 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.043709 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-64j7z_06b6ce26-951d-44da-ae1d-4b20971557fd/kube-rbac-proxy-self/0.log" Sep 07 13:36:51.103290 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.103244 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7c797d6c-2sxp6_e7fa1752-9763-4427-945a-81a822d186bd/metrics-server/0.log" Sep 07 13:36:51.149688 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.149659 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-6zcxv_10ae1775-1aa5-4079-b465-4945352e2cde/monitoring-plugin/0.log" Sep 07 13:36:51.224654 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.224561 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7lsw4_22a97db3-4fcb-4e13-b1cc-54ffd9845248/node-exporter/0.log" Sep 07 13:36:51.296660 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.296627 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7lsw4_22a97db3-4fcb-4e13-b1cc-54ffd9845248/kube-rbac-proxy/0.log" Sep 07 13:36:51.332406 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.332375 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7lsw4_22a97db3-4fcb-4e13-b1cc-54ffd9845248/init-textfile/0.log" Sep 07 13:36:51.689550 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.689512 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40429edf-7358-4009-8a8e-76aa05defb1b/prometheus/0.log" Sep 07 13:36:51.728247 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.728168 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40429edf-7358-4009-8a8e-76aa05defb1b/config-reloader/0.log" Sep 07 13:36:51.769998 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.769950 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40429edf-7358-4009-8a8e-76aa05defb1b/thanos-sidecar/0.log" Sep 07 13:36:51.808890 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.808841 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40429edf-7358-4009-8a8e-76aa05defb1b/kube-rbac-proxy-web/0.log" Sep 07 13:36:51.837310 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.837278 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40429edf-7358-4009-8a8e-76aa05defb1b/kube-rbac-proxy/0.log" Sep 07 13:36:51.865312 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.865277 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40429edf-7358-4009-8a8e-76aa05defb1b/kube-rbac-proxy-thanos/0.log" Sep 07 13:36:51.906362 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.906294 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40429edf-7358-4009-8a8e-76aa05defb1b/init-config-reloader/0.log" Sep 07 13:36:51.949083 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.948968 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-n65rb_bfbbf363-2a82-4f30-88a0-c28dff487e42/prometheus-operator/0.log" Sep 07 13:36:51.975532 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:51.975505 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-n65rb_bfbbf363-2a82-4f30-88a0-c28dff487e42/kube-rbac-proxy/0.log" Sep 07 13:36:52.178306 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:52.178257 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f656656f-7r4lc_aee2d72e-4493-453b-9be4-5b60e710cd2c/thanos-query/0.log" Sep 07 13:36:52.206089 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:52.205986 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f656656f-7r4lc_aee2d72e-4493-453b-9be4-5b60e710cd2c/kube-rbac-proxy-web/0.log" Sep 07 13:36:52.234714 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:52.234687 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f656656f-7r4lc_aee2d72e-4493-453b-9be4-5b60e710cd2c/kube-rbac-proxy/0.log" Sep 07 13:36:52.270250 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:52.270221 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f656656f-7r4lc_aee2d72e-4493-453b-9be4-5b60e710cd2c/prom-label-proxy/0.log" Sep 07 13:36:52.302538 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:52.302509 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f656656f-7r4lc_aee2d72e-4493-453b-9be4-5b60e710cd2c/kube-rbac-proxy-rules/0.log" Sep 07 13:36:52.346777 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:52.346742 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f656656f-7r4lc_aee2d72e-4493-453b-9be4-5b60e710cd2c/kube-rbac-proxy-metrics/0.log" Sep 07 13:36:53.596386 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:53.596353 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-kwlqf_974b5145-6856-4fea-b95d-e393bd76baed/networking-console-plugin/0.log" Sep 07 13:36:53.755462 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:53.755423 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt"] Sep 07 13:36:53.759309 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:53.759284 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:53.762697 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:53.762677 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-scn9z\"/\"default-dockercfg-wv848\"" Sep 07 13:36:53.768950 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:53.768924 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt"] Sep 07 13:36:53.908693 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:53.908650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-sys\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:53.908893 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:53.908720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-proc\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:53.908893 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:53.908781 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-lib-modules\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:53.908893 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:53.908806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-podres\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:53.908893 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:53.908827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xrk5\" (UniqueName: \"kubernetes.io/projected/e6eb5789-3b20-4546-b668-dcdf61373303-kube-api-access-6xrk5\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.010357 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.010316 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-proc\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.010557 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.010393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-lib-modules\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.010557 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.010413 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-podres\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.010557 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.010434 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xrk5\" (UniqueName: \"kubernetes.io/projected/e6eb5789-3b20-4546-b668-dcdf61373303-kube-api-access-6xrk5\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.010557 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.010530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-sys\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.010811 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.010583 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-lib-modules\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.010811 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.010635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-sys\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.010811 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.010670 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-proc\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.010811 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.010718 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e6eb5789-3b20-4546-b668-dcdf61373303-podres\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.022006 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.021967 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xrk5\" (UniqueName: \"kubernetes.io/projected/e6eb5789-3b20-4546-b668-dcdf61373303-kube-api-access-6xrk5\") pod \"perf-node-gather-daemonset-4jpnt\" (UID: \"e6eb5789-3b20-4546-b668-dcdf61373303\") " pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.077017 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.076766 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.394732 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.394694 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt"] Sep 07 13:36:54.396150 ip-10-0-142-216 kubenswrapper[2565]: W0907 13:36:54.396116 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode6eb5789_3b20_4546_b668_dcdf61373303.slice/crio-ac706928d29e9bf0015faac970feaa0391fc7151f77b051d6462895e00eab913 WatchSource:0}: Error finding container ac706928d29e9bf0015faac970feaa0391fc7151f77b051d6462895e00eab913: Status 404 returned error can't find the container with id ac706928d29e9bf0015faac970feaa0391fc7151f77b051d6462895e00eab913 Sep 07 13:36:54.878313 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.878265 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" event={"ID":"e6eb5789-3b20-4546-b668-dcdf61373303","Type":"ContainerStarted","Data":"7cc93987651780cbe39e057ad087bcf717083994a0b830365bce28556be8bdbc"} Sep 07 13:36:54.878973 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.878332 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" event={"ID":"e6eb5789-3b20-4546-b668-dcdf61373303","Type":"ContainerStarted","Data":"ac706928d29e9bf0015faac970feaa0391fc7151f77b051d6462895e00eab913"} Sep 07 13:36:54.878973 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.878360 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:36:54.884002 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.883967 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-s42nz_1edf6720-0ac6-4149-8bee-e617b17c9613/download-server/0.log" Sep 07 13:36:54.904726 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:54.904667 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" podStartSLOduration=1.904646091 podStartE2EDuration="1.904646091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 13:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 13:36:54.902017273 +0000 UTC m=+2236.582403044" watchObservedRunningTime="2026-09-07 13:36:54.904646091 +0000 UTC m=+2236.585031862" Sep 07 13:36:56.806371 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:56.806337 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xg2sx_65861502-da2f-4c4d-a629-c860d4c3f6e9/dns/0.log" Sep 07 13:36:56.834175 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:56.834134 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xg2sx_65861502-da2f-4c4d-a629-c860d4c3f6e9/kube-rbac-proxy/0.log" Sep 07 13:36:56.929192 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:56.929164 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vswfs_2fddfa4d-1664-4139-8891-e7c6bc67fde4/dns-node-resolver/0.log" Sep 07 13:36:57.524484 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:57.524451 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65cccd88c9-q57vf_34d5e8f1-56b0-4169-86e0-7418b9093a50/registry/0.log" Sep 07 13:36:57.549853 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:57.549824 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hjp2h_9e73d458-8cb5-4913-9ad2-54f0615ef55d/node-ca/0.log" Sep 07 13:36:59.261883 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:59.261847 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546b89bbb7-spq8c_2fc20259-3152-445f-b813-f6dc5da42ed1/kube-auth-proxy/0.log" Sep 07 13:36:59.359885 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:36:59.359847 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6bld5_6009be81-7b8d-44b4-8fa4-d8c3da6736cc/payload-pre-processing/0.log" Sep 07 13:37:00.372414 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:00.372386 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2p52x_d48d2f85-2ba1-4b88-9f3d-f512294a07f3/serve-healthcheck-canary/0.log" Sep 07 13:37:00.905413 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:00.905384 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-scn9z/perf-node-gather-daemonset-4jpnt" Sep 07 13:37:01.190041 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:01.189957 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wj79l_852d1deb-e626-454d-a9b0-19759119b16d/kube-rbac-proxy/0.log" Sep 07 13:37:01.219289 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:01.219256 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wj79l_852d1deb-e626-454d-a9b0-19759119b16d/exporter/0.log" Sep 07 13:37:01.249370 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:01.249342 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wj79l_852d1deb-e626-454d-a9b0-19759119b16d/extractor/0.log" Sep 07 13:37:04.633913 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:04.633879 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546b89bbb7-spq8c_2fc20259-3152-445f-b813-f6dc5da42ed1/kube-auth-proxy/0.log" Sep 07 13:37:04.698107 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:04.698065 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5545ff49d6-j24tl_2eae9c0e-b88a-4a38-9a5c-e58c2c744c38/manager/0.log" Sep 07 13:37:04.749525 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:04.749491 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6bld5_6009be81-7b8d-44b4-8fa4-d8c3da6736cc/payload-pre-processing/0.log" Sep 07 13:37:04.754611 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:04.754578 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5545ff49d6-j24tl_2eae9c0e-b88a-4a38-9a5c-e58c2c744c38/copy-manifests/0.log" Sep 07 13:37:06.236471 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:06.236438 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58cbfb6566-4dcqc_d6d21826-fdba-448d-bb2f-4c20a3a1861c/manager/0.log" Sep 07 13:37:07.014435 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:07.014406 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-546b89bbb7-spq8c_2fc20259-3152-445f-b813-f6dc5da42ed1/kube-auth-proxy/0.log" Sep 07 13:37:07.092944 ip-10-0-142-216 kubenswrapper[2565]: I0907 13:37:07.092910 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6bld5_6009be81-7b8d-44b4-8fa4-d8c3da6736cc/payload-pre-processing/0.log"