Sep 10 04:39:23.849501 ip-10-0-136-213 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 04:39:23.849514 ip-10-0-136-213 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 04:39:23.849524 ip-10-0-136-213 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 04:39:23.849863 ip-10-0-136-213 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 04:39:34.018535 ip-10-0-136-213 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 04:39:34.018554 ip-10-0-136-213 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a542ee092d8742228caccbc6c75c144d -- Sep 10 04:41:19.154189 ip-10-0-136-213 systemd[1]: Starting Kubernetes Kubelet... Sep 10 04:41:19.597496 ip-10-0-136-213 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 04:41:19.597496 ip-10-0-136-213 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 04:41:19.597496 ip-10-0-136-213 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 04:41:19.597496 ip-10-0-136-213 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 04:41:19.597496 ip-10-0-136-213 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 04:41:19.598175 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.597542 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 04:41:19.601760 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601742 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 04:41:19.601760 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601759 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601763 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601766 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601769 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601773 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601776 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601778 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601781 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601784 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601787 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601793 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601796 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601814 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601821 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601827 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601830 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601834 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601836 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601840 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601842 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 04:41:19.601847 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601845 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601848 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601850 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601853 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601856 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601859 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601862 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601865 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601867 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601870 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601872 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601875 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601877 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601880 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601883 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601885 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601888 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601891 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601894 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 04:41:19.602335 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601896 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601899 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601908 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601912 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601916 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601919 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601921 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601924 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601926 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601929 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601932 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601934 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601937 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601940 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601942 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601946 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601949 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601953 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601957 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601960 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 04:41:19.602821 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601962 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601965 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601967 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601971 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601973 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601976 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601978 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601981 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601985 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601988 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601991 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601994 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.601998 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602001 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602003 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602006 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602008 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602011 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602014 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 04:41:19.603309 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602017 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602019 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602022 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602024 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602027 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602030 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602033 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602035 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602038 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602041 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602044 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602047 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602049 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602052 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602056 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602058 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602061 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602063 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602066 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 04:41:19.603817 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602069 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602770 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602776 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602779 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602782 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602785 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602788 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602791 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602794 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602797 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602814 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602817 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602820 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602823 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602826 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602828 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602831 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602834 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602836 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 04:41:19.604293 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602840 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602842 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602845 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602848 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602850 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602853 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602856 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602858 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602861 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602863 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602866 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602868 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602870 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602875 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602879 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602882 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602885 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602887 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602890 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 04:41:19.604767 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602893 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602896 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602899 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602902 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602905 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602908 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602911 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602914 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602916 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602919 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602923 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602926 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602929 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602932 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602935 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602937 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602941 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602943 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602946 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602949 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 04:41:19.605252 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602951 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602954 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602957 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602959 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602962 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602964 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602967 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602971 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602973 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602975 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602979 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602982 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602984 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602987 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602990 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602992 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602995 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.602997 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603003 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603005 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 04:41:19.605750 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603008 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603011 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603013 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603016 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603019 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603022 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603025 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603027 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603030 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603034 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603038 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603040 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603044 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603046 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603049 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603052 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603055 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603058 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603060 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603063 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 04:41:19.606261 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603065 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603068 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603449 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603454 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603457 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603460 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603463 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603466 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603469 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603471 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603475 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603478 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603480 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603483 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603486 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603489 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603491 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603496 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603499 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 04:41:19.606745 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603503 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603505 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603508 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603512 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603515 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603518 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603521 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603524 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603528 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603530 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603533 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603536 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603538 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603541 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603543 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603546 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603548 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603551 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603554 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603556 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 04:41:19.607226 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603560 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603562 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603565 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603569 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603572 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603574 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603577 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603579 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603582 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603584 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603587 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603590 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603593 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603595 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603598 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603601 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603604 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603606 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603609 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603611 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 04:41:19.607718 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603614 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603617 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603619 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603622 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603624 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603627 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603629 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603632 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603634 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603637 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603639 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603642 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603644 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603647 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603650 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603654 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603657 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603660 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603663 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603665 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 04:41:19.608211 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603668 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603670 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603673 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603676 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603678 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603681 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603685 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603688 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603691 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603694 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603697 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603699 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603702 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603705 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603707 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603710 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603712 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603715 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603717 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 04:41:19.608727 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603720 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603722 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.603725 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603813 2572 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603846 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603853 2572 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603857 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603862 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603866 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603871 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603875 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603878 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603881 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603884 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603888 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603892 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603895 2572 flags.go:64] FLAG: --cgroup-root="" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603898 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603900 2572 flags.go:64] FLAG: --client-ca-file="" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603903 2572 flags.go:64] FLAG: --cloud-provider="external" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603906 2572 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603910 2572 flags.go:64] FLAG: --cluster-domain="" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603913 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603916 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 04:41:19.609237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603919 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603922 2572 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603926 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603929 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603932 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603935 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603938 2572 flags.go:64] FLAG: --contention-profiling="false" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603941 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603943 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603946 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603949 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603958 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603961 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603963 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603966 2572 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603969 2572 flags.go:64] FLAG: --enable-server="true" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603973 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603979 2572 flags.go:64] FLAG: --event-burst="100" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603982 2572 flags.go:64] FLAG: --event-qps="50" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603985 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603988 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603991 2572 flags.go:64] FLAG: --eviction-hard="" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603995 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.603998 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604001 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 04:41:19.609918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604004 2572 flags.go:64] FLAG: --eviction-soft="" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604007 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604011 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604014 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604018 2572 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604021 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604024 2572 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604027 2572 flags.go:64] FLAG: --feature-gates="" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604031 2572 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604034 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604037 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604040 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604043 2572 flags.go:64] FLAG: --healthz-port="10248" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604046 2572 flags.go:64] FLAG: --help="false" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604049 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-136-213.ec2.internal" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604052 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604055 2572 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604057 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604061 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604064 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604067 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604071 2572 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604074 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 04:41:19.610524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604078 2572 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604082 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604085 2572 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604088 2572 flags.go:64] FLAG: --kube-reserved="" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604090 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604093 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604097 2572 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604100 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604102 2572 flags.go:64] FLAG: --lock-file="" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604105 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604108 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604111 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604117 2572 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604120 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604122 2572 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604125 2572 flags.go:64] FLAG: --logging-format="text" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604128 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604131 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604134 2572 flags.go:64] FLAG: --manifest-url="" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604137 2572 flags.go:64] FLAG: --manifest-url-header="" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604141 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604144 2572 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604148 2572 flags.go:64] FLAG: --max-pods="110" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604151 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604154 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 04:41:19.611187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604157 2572 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604160 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604163 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604166 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604169 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604177 2572 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604180 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604183 2572 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604187 2572 flags.go:64] FLAG: --pod-cidr="" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604190 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604196 2572 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604199 2572 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604202 2572 flags.go:64] FLAG: --pods-per-core="0" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604205 2572 flags.go:64] FLAG: --port="10250" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604208 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604210 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-014690939f93c84d7" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604214 2572 flags.go:64] FLAG: --qos-reserved="" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604216 2572 flags.go:64] FLAG: --read-only-port="10255" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604219 2572 flags.go:64] FLAG: --register-node="true" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604222 2572 flags.go:64] FLAG: --register-with-taints="" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604225 2572 flags.go:64] FLAG: --registry-burst="10" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604228 2572 flags.go:64] FLAG: --registry-qps="5" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604231 2572 flags.go:64] FLAG: --reserved-cpus="" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604234 2572 flags.go:64] FLAG: --reserved-memory="" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604237 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 04:41:19.611866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604240 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604243 2572 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604246 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604248 2572 flags.go:64] FLAG: --runonce="false" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604251 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604254 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604257 2572 flags.go:64] FLAG: --seccomp-default="false" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604259 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604262 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604265 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604268 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604271 2572 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604275 2572 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604278 2572 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604281 2572 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604285 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604288 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604291 2572 flags.go:64] FLAG: --system-cgroups="" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604294 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604299 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604302 2572 flags.go:64] FLAG: --tls-cert-file="" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604305 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604309 2572 flags.go:64] FLAG: --tls-min-version="" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604312 2572 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604315 2572 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 04:41:19.612596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604318 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604320 2572 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604323 2572 flags.go:64] FLAG: --v="2" Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604327 2572 flags.go:64] FLAG: --version="false" Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604331 2572 flags.go:64] FLAG: --vmodule="" Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604334 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.604338 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604448 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604451 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604454 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604457 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604459 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604462 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604464 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604467 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604470 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604473 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604475 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604478 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604481 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604483 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604486 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 04:41:19.613316 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604490 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604492 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604495 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604498 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604501 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604504 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604506 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604509 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604511 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604514 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604516 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604519 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604522 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604524 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604526 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604531 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604534 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604537 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604540 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604543 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 04:41:19.613944 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604547 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604550 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604552 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604555 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604558 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604560 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604563 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604566 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604569 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604572 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604575 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604577 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604581 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604584 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604587 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604589 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604593 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604596 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604599 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 04:41:19.614506 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604601 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604604 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604606 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604609 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604612 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604615 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604619 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604622 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604625 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604628 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604630 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604633 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604635 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604638 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604642 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604646 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604649 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604652 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604655 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 04:41:19.615322 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604657 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604660 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604663 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604666 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604668 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604671 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604675 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604677 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604680 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604682 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604685 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604688 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604691 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604693 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604696 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604698 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604701 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604704 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604706 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604709 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 04:41:19.616016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604711 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604714 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604716 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604719 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604721 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.604724 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.605466 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.613569 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.613588 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.613605 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.613617 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613682 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613687 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613691 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613694 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 04:41:19.616518 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613697 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613699 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613704 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613708 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613711 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613714 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613717 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613720 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613722 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613725 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613729 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613731 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613733 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613736 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613739 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613741 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613743 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613746 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613749 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 04:41:19.617048 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613752 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613756 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613759 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613762 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613765 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613767 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613776 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613779 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613782 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613784 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613787 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613789 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613792 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613795 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613797 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613811 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613814 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613817 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613820 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613822 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 04:41:19.617509 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613825 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613828 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613830 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613833 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613836 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613838 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613840 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613843 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613846 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613848 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613851 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613853 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613856 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613859 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613861 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613863 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613867 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613870 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613872 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613881 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 04:41:19.618016 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613884 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613886 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613889 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613891 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613894 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613896 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613899 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613901 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613904 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613906 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613909 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613912 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613914 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613917 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613919 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613922 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613924 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613926 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613929 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613931 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 04:41:19.618523 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613933 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613936 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613939 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613941 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613944 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613946 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613949 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613951 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613954 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613958 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613963 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613966 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613978 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613981 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613983 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 04:41:19.619049 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.613986 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.613991 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614137 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614142 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614145 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614148 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614151 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614154 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614157 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614160 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614162 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614165 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614167 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614170 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614172 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 04:41:19.619459 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614175 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614177 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614180 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614183 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614186 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614188 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614191 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614194 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614196 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614199 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614201 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614204 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614207 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614210 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614212 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614224 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614226 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614229 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614231 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614234 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 04:41:19.619862 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614236 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614239 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614241 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614244 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614246 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614248 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614251 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614254 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614256 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614259 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614263 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614266 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614268 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614271 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614274 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614276 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614279 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614281 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614302 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 04:41:19.620364 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614307 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614310 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614313 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614316 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614318 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614321 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614324 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614326 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614329 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614338 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614340 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614343 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614345 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614348 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614351 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614353 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614356 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614358 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614360 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 04:41:19.620838 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614363 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614366 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614368 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614371 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614373 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614376 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614379 2572 feature_gate.go:328] unrecognized feature gate: Example Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614381 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614384 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614386 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614389 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614392 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614402 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614406 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614409 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614412 2572 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614415 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614417 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614420 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 04:41:19.621312 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614423 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614425 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614428 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614437 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614445 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614448 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614450 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614453 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:19.614455 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.614460 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.615396 2572 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.618842 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.619956 2572 server.go:1019] "Starting client certificate rotation" Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.620050 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 04:41:19.621768 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.620079 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 04:41:19.622900 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.622885 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 04:41:19.624061 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.624043 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 04:41:19.636084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.636063 2572 log.go:25] "Validated CRI v1 runtime API" Sep 10 04:41:19.641018 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.640994 2572 log.go:25] "Validated CRI v1 image API" Sep 10 04:41:19.642165 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.642150 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 04:41:19.645536 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.645514 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81090128-588a-46d8-af1c-fff8016eea5e:/dev/nvme0n1p3 a087446c-73c4-4a00-92f6-0888e6778fbd:/dev/nvme0n1p4] Sep 10 04:41:19.645623 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.645535 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 04:41:19.649633 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.649612 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 04:41:19.651857 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.651728 2572 manager.go:216] Machine: {Timestamp:2026-09-10 04:41:19.650310036 +0000 UTC m=+0.385788489 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099722 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26c36c78ba31e7f8e74764c68cbbeb SystemUUID:ec26c36c-78ba-31e7-f8e7-4764c68cbbeb BootID:a542ee09-2d87-4222-8cac-cbc6c75c144d Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:76:8e:19:ab:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:76:8e:19:ab:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:99:97:57:ed:e4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 04:41:19.651857 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.651847 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 04:41:19.652016 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.651965 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 04:41:19.655420 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.655389 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 04:41:19.655559 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.655423 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-213.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 04:41:19.656248 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.656237 2572 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 04:41:19.656299 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.656251 2572 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 04:41:19.656299 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.656259 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 04:41:19.656299 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.656274 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 04:41:19.658203 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.658190 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 04:41:19.662887 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.662869 2572 state_mem.go:36] "Initialized new in-memory state store" Sep 10 04:41:19.663014 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.663004 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 04:41:19.663047 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.663035 2572 kubelet.go:486] "Attempting to sync node with API server" Sep 10 04:41:19.663047 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.663046 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 04:41:19.663105 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.663060 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 04:41:19.663105 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.663070 2572 kubelet.go:398] "Adding apiserver pod source" Sep 10 04:41:19.663105 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.663085 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 04:41:19.663501 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.663479 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 04:41:19.663501 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.663503 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 04:41:19.666797 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.666764 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 04:41:19.669204 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.669186 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 04:41:19.669289 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.669216 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 04:41:19.669430 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.669415 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 04:41:19.669467 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.669435 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 04:41:19.669467 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.669444 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 04:41:19.669467 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.669453 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 04:41:19.669467 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.669463 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 04:41:19.669467 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.669468 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 04:41:19.670070 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.670056 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 04:41:19.670108 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.670072 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 04:41:19.670108 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.670082 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 04:41:19.670108 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.670091 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 04:41:19.670108 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.670103 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 04:41:19.670218 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.670112 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 04:41:19.670961 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.670950 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 04:41:19.671014 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.670963 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 04:41:19.674264 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.674242 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-213.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 04:41:19.674355 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.674278 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-213.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 04:41:19.674355 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.674334 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 04:41:19.675084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.675074 2572 server.go:1268] "Started kubelet" Sep 10 04:41:19.675214 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.675159 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 04:41:19.675318 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.675194 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 04:41:19.675318 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.675261 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 04:41:19.676091 ip-10-0-136-213 systemd[1]: Started Kubernetes Kubelet. Sep 10 04:41:19.676843 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.676824 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 04:41:19.677266 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.677235 2572 server.go:310] "Adding debug handlers to kubelet server" Sep 10 04:41:19.677370 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.677290 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c64l5" Sep 10 04:41:19.680587 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.680559 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 04:41:19.682415 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.682390 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c64l5" Sep 10 04:41:19.685333 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.684197 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-213.ec2.internal.18d3dbeb12c6f68b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-213.ec2.internal,UID:ip-10-0-136-213.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-213.ec2.internal,},FirstTimestamp:2026-09-10 04:41:19.675045515 +0000 UTC m=+0.410523967,LastTimestamp:2026-09-10 04:41:19.675045515 +0000 UTC m=+0.410523967,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-213.ec2.internal,}" Sep 10 04:41:19.685915 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.685896 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 04:41:19.685998 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.685916 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 04:41:19.686184 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.686164 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:19.686628 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.686613 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 04:41:19.686831 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.686819 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 04:41:19.686995 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.686625 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 04:41:19.687085 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.687009 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 04:41:19.687184 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.687091 2572 factory.go:55] Registering systemd factory Sep 10 04:41:19.687184 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.687105 2572 factory.go:223] Registration of the systemd container factory successfully Sep 10 04:41:19.687184 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.687118 2572 reconstruct.go:97] "Volume reconstruction finished" Sep 10 04:41:19.687184 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.687128 2572 reconciler.go:29] "Reconciler: start to sync state" Sep 10 04:41:19.687528 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.687506 2572 factory.go:153] Registering CRI-O factory Sep 10 04:41:19.687528 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.687528 2572 factory.go:223] Registration of the crio container factory successfully Sep 10 04:41:19.687705 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.687546 2572 factory.go:103] Registering Raw factory Sep 10 04:41:19.687705 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.687560 2572 manager.go:1196] Started watching for new ooms in manager Sep 10 04:41:19.688029 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.688015 2572 manager.go:349] Starting recovery of all containers Sep 10 04:41:19.689335 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.689306 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-213.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 04:41:19.689460 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.689439 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 04:41:19.698911 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.698695 2572 manager.go:354] Recovery completed Sep 10 04:41:19.704675 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.704470 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 04:41:19.706459 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.706439 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientMemory" Sep 10 04:41:19.706553 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.706473 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 04:41:19.706553 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.706483 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientPID" Sep 10 04:41:19.707043 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.707027 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 04:41:19.707043 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.707042 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 04:41:19.707146 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.707045 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 04:41:19.707146 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.707061 2572 state_mem.go:36] "Initialized new in-memory state store" Sep 10 04:41:19.707790 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.707766 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientMemory" Sep 10 04:41:19.707872 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.707815 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 04:41:19.707872 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.707831 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientPID" Sep 10 04:41:19.709174 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.709160 2572 policy_none.go:49] "None policy: Start" Sep 10 04:41:19.709217 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.709180 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 04:41:19.709217 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.709193 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 04:41:19.711206 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.711194 2572 policy_none.go:47] "Start" Sep 10 04:41:19.772468 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.753618 2572 manager.go:337] "Starting Device Plugin manager" Sep 10 04:41:19.772468 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.753664 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 04:41:19.772468 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.753675 2572 server.go:87] "Starting device plugin registration server" Sep 10 04:41:19.772468 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.754120 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 04:41:19.772468 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.754132 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 04:41:19.772468 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.754221 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 04:41:19.772468 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.754316 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 04:41:19.772468 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.754325 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 04:41:19.772468 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.755006 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 04:41:19.772468 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.755057 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:19.796998 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.796961 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 04:41:19.798203 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.798184 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 04:41:19.798272 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.798259 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 04:41:19.798322 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.798299 2572 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 04:41:19.798432 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.798413 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 04:41:19.800327 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.800299 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 04:41:19.855367 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.855250 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 04:41:19.856408 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.856386 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientMemory" Sep 10 04:41:19.856528 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.856420 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 04:41:19.856528 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.856431 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientPID" Sep 10 04:41:19.856528 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.856454 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-213.ec2.internal" Sep 10 04:41:19.864095 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.864070 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-213.ec2.internal" Sep 10 04:41:19.864209 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.864102 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-213.ec2.internal\": node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:19.879624 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.879596 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:19.898506 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.898457 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-213.ec2.internal"] Sep 10 04:41:19.898688 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.898544 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 04:41:19.899542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.899525 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientMemory" Sep 10 04:41:19.899634 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.899559 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 04:41:19.899634 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.899573 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientPID" Sep 10 04:41:19.901940 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.901925 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 04:41:19.902055 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.902040 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" Sep 10 04:41:19.902095 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.902069 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 04:41:19.902702 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.902686 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientMemory" Sep 10 04:41:19.902797 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.902718 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 04:41:19.902797 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.902727 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientMemory" Sep 10 04:41:19.902797 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.902754 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 04:41:19.902797 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.902778 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientPID" Sep 10 04:41:19.902797 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.902732 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientPID" Sep 10 04:41:19.905172 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.905154 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-213.ec2.internal" Sep 10 04:41:19.905263 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.905179 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 04:41:19.905899 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.905881 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientMemory" Sep 10 04:41:19.905976 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.905908 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 04:41:19.905976 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.905918 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientPID" Sep 10 04:41:19.928819 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.928782 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-213.ec2.internal\" not found" node="ip-10-0-136-213.ec2.internal" Sep 10 04:41:19.933506 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.933489 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-213.ec2.internal\" not found" node="ip-10-0-136-213.ec2.internal" Sep 10 04:41:19.979872 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:19.979842 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:19.989623 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.989594 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f78df6d50d8bd4ba0dec82fb21a37c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal\" (UID: \"2f78df6d50d8bd4ba0dec82fb21a37c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" Sep 10 04:41:19.989714 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.989629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee62c14ad64bc1c5c73f7816f7b32bcc-config\") pod \"kube-apiserver-proxy-ip-10-0-136-213.ec2.internal\" (UID: \"ee62c14ad64bc1c5c73f7816f7b32bcc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-213.ec2.internal" Sep 10 04:41:19.989714 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:19.989644 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f78df6d50d8bd4ba0dec82fb21a37c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal\" (UID: \"2f78df6d50d8bd4ba0dec82fb21a37c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" Sep 10 04:41:20.079943 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:20.079907 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:20.090333 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:20.090303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f78df6d50d8bd4ba0dec82fb21a37c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal\" (UID: \"2f78df6d50d8bd4ba0dec82fb21a37c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" Sep 10 04:41:20.090416 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:20.090338 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f78df6d50d8bd4ba0dec82fb21a37c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal\" (UID: \"2f78df6d50d8bd4ba0dec82fb21a37c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" Sep 10 04:41:20.090416 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:20.090365 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee62c14ad64bc1c5c73f7816f7b32bcc-config\") pod \"kube-apiserver-proxy-ip-10-0-136-213.ec2.internal\" (UID: \"ee62c14ad64bc1c5c73f7816f7b32bcc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-213.ec2.internal" Sep 10 04:41:20.090416 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:20.090404 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2f78df6d50d8bd4ba0dec82fb21a37c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal\" (UID: \"2f78df6d50d8bd4ba0dec82fb21a37c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" Sep 10 04:41:20.090518 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:20.090420 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f78df6d50d8bd4ba0dec82fb21a37c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal\" (UID: \"2f78df6d50d8bd4ba0dec82fb21a37c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" Sep 10 04:41:20.090518 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:20.090404 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ee62c14ad64bc1c5c73f7816f7b32bcc-config\") pod \"kube-apiserver-proxy-ip-10-0-136-213.ec2.internal\" (UID: \"ee62c14ad64bc1c5c73f7816f7b32bcc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-213.ec2.internal" Sep 10 04:41:20.180841 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:20.180750 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:21.462892 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:21.462872 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" Sep 10 04:41:21.468281 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:21.467770 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-213.ec2.internal" Sep 10 04:41:21.512809 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:21.512771 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:21.613345 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:21.613320 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:21.714007 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:21.713923 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:21.814522 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:21.814489 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:21.851052 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:21.851019 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 04:41:21.851508 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:21.851163 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 04:41:21.914726 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:21.914696 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:21.920176 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:21.920137 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 04:36:20 +0000 UTC" deadline="2028-06-08 12:27:16.274479248 +0000 UTC" Sep 10 04:41:21.920176 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:21.920176 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15295h45m54.354306988s" Sep 10 04:41:22.014883 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:22.014802 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-213.ec2.internal\" not found" Sep 10 04:41:22.054800 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.054770 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 04:41:22.067177 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:22.067145 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee62c14ad64bc1c5c73f7816f7b32bcc.slice/crio-a5e2eb3af8f838333a974ba945bdeceedd401a1728b2e41870391b4b6cf634a7 WatchSource:0}: Error finding container a5e2eb3af8f838333a974ba945bdeceedd401a1728b2e41870391b4b6cf634a7: Status 404 returned error can't find the container with id a5e2eb3af8f838333a974ba945bdeceedd401a1728b2e41870391b4b6cf634a7 Sep 10 04:41:22.067939 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:22.067908 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f78df6d50d8bd4ba0dec82fb21a37c8.slice/crio-3a476d355567c2c77ed47c202b84cb54030fa51e1ea09ca18246da18f3c43179 WatchSource:0}: Error finding container 3a476d355567c2c77ed47c202b84cb54030fa51e1ea09ca18246da18f3c43179: Status 404 returned error can't find the container with id 3a476d355567c2c77ed47c202b84cb54030fa51e1ea09ca18246da18f3c43179 Sep 10 04:41:22.071812 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.071795 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 04:41:22.118771 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.118740 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-213.ec2.internal" Sep 10 04:41:22.122917 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.122895 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 04:41:22.126490 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.126473 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 04:41:22.127579 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.127564 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" Sep 10 04:41:22.138894 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.138871 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 04:41:22.303748 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.303664 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 04:41:22.760818 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.760743 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 04:41:22.895969 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.895933 2572 apiserver.go:52] "Watching apiserver" Sep 10 04:41:22.906565 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.906532 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 04:41:22.907081 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.907056 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-n2dx5","openshift-network-operator/iptables-alerter-9drwx","openshift-ovn-kubernetes/ovnkube-node-q7lb4","kube-system/konnectivity-agent-ltxkf","openshift-cluster-node-tuning-operator/tuned-wkj57","openshift-dns/node-resolver-z2nhq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal","openshift-multus/multus-grjvl","openshift-network-diagnostics/network-check-target-sgwg2","kube-system/global-pull-secret-syncer-mh4x5","kube-system/kube-apiserver-proxy-ip-10-0-136-213.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz","openshift-image-registry/node-ca-clkch","openshift-multus/multus-additional-cni-plugins-sg5jd"] Sep 10 04:41:22.911780 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.911755 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:22.911917 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:22.911862 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:22.914028 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.914004 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-clkch" Sep 10 04:41:22.917387 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.917033 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.918357 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.917710 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 04:41:22.918357 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.917763 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 04:41:22.918357 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.918109 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 04:41:22.918357 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.918170 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 04:41:22.918602 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.918472 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7c7zc\"" Sep 10 04:41:22.919695 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.919672 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 04:41:22.919808 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.919682 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 04:41:22.920124 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.920109 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 04:41:22.920210 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.920193 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 04:41:22.920365 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.920345 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j4jxp\"" Sep 10 04:41:22.920421 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.920387 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 04:41:22.920421 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.920387 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 04:41:22.920630 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.920617 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:22.923016 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.922997 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 04:41:22.923383 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.923364 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:22.923485 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.923463 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vhk75\"" Sep 10 04:41:22.924117 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.923374 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 04:41:22.925882 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.925854 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 04:41:22.926116 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.926101 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7k27c\"" Sep 10 04:41:22.926597 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.926562 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 04:41:22.926675 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.926565 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z2nhq" Sep 10 04:41:22.928824 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.928665 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:22.928824 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:22.928731 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:22.929044 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.929017 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 04:41:22.929366 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.929348 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m66hn\"" Sep 10 04:41:22.929996 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.929935 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 04:41:22.931284 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.931261 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 04:41:22.931598 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.931581 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:22.931971 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.931948 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-node-log\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932055 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.931987 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-ovn-node-metrics-cert\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932055 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932012 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msf64\" (UniqueName: \"kubernetes.io/projected/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-kube-api-access-msf64\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932055 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932036 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-run-ovn\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932207 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932058 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-cni-netd\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932207 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932082 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-env-overrides\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932207 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932123 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:22.932207 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932147 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b773f2a8-5f9d-4a6f-850f-3c37ae6f4705-host\") pod \"node-ca-clkch\" (UID: \"b773f2a8-5f9d-4a6f-850f-3c37ae6f4705\") " pod="openshift-image-registry/node-ca-clkch" Sep 10 04:41:22.932207 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932170 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-run-netns\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932207 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932198 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-run-systemd\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932566 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932221 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932566 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932251 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-log-socket\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932566 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932318 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-cni-bin\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932566 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932359 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e-agent-certs\") pod \"konnectivity-agent-ltxkf\" (UID: \"c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e\") " pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:22.932566 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932384 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46bf0532-232f-423a-add2-7c0ff0a5a9a9-dbus\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:22.932566 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932405 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-slash\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932566 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932468 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-run-openvswitch\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932566 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932501 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-ovnkube-config\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932566 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932524 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-ovnkube-script-lib\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.932566 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932550 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.933020 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932573 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e-konnectivity-ca\") pod \"konnectivity-agent-ltxkf\" (UID: \"c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e\") " pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:22.933020 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932595 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b773f2a8-5f9d-4a6f-850f-3c37ae6f4705-serviceca\") pod \"node-ca-clkch\" (UID: \"b773f2a8-5f9d-4a6f-850f-3c37ae6f4705\") " pod="openshift-image-registry/node-ca-clkch" Sep 10 04:41:22.933020 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932633 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4zm\" (UniqueName: \"kubernetes.io/projected/b773f2a8-5f9d-4a6f-850f-3c37ae6f4705-kube-api-access-wc4zm\") pod \"node-ca-clkch\" (UID: \"b773f2a8-5f9d-4a6f-850f-3c37ae6f4705\") " pod="openshift-image-registry/node-ca-clkch" Sep 10 04:41:22.933020 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932681 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-kubelet\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.933020 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932709 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-var-lib-openvswitch\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.933020 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932739 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46bf0532-232f-423a-add2-7c0ff0a5a9a9-kubelet-config\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:22.933020 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932767 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-systemd-units\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.933020 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.932789 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-etc-openvswitch\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:22.933777 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.933756 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 04:41:22.933976 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.933959 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s677c\"" Sep 10 04:41:22.934168 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.934155 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 04:41:22.934222 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.934167 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 04:41:22.934380 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.934365 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 04:41:22.934581 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.934566 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 04:41:22.934782 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.934767 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9drwx" Sep 10 04:41:22.936139 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.936123 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:22.936215 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:22.936188 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:22.936822 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.936803 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 04:41:22.937930 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.937268 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hwcjp\"" Sep 10 04:41:22.938176 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.938157 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 04:41:22.939081 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.939064 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 04:41:22.940229 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.940214 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-grjvl" Sep 10 04:41:22.943019 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.942999 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 04:41:22.943800 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.943486 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdnb2\"" Sep 10 04:41:22.943800 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.943737 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:22.947044 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.947021 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cqf2m\"" Sep 10 04:41:22.947294 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.947196 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 04:41:22.947435 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.947352 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 04:41:22.947586 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.947514 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 04:41:22.953303 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.953285 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7527r" Sep 10 04:41:22.965985 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:22.965962 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7527r" Sep 10 04:41:23.021020 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.020942 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.032976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-log-socket\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-cni-bin\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-lib-modules\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033073 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/195cb462-b600-46bc-ab74-21a87101050a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033098 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e9259fd-d4c3-49ad-8d0b-4cc695654363-host-slash\") pod \"iptables-alerter-9drwx\" (UID: \"8e9259fd-d4c3-49ad-8d0b-4cc695654363\") " pod="openshift-network-operator/iptables-alerter-9drwx" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033141 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46bf0532-232f-423a-add2-7c0ff0a5a9a9-dbus\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-slash\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033218 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46bf0532-232f-423a-add2-7c0ff0a5a9a9-dbus\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033229 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-run-openvswitch\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033269 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-slash\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033272 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-log-socket\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033298 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-run-openvswitch\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033308 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-cni-bin\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033302 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-ovnkube-script-lib\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033345 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e2a92d6-a5fb-4091-802d-e1a942c45416-tmp\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033375 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/195cb462-b600-46bc-ab74-21a87101050a-system-cni-dir\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.034769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033401 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcqzj\" (UniqueName: \"kubernetes.io/projected/195cb462-b600-46bc-ab74-21a87101050a-kube-api-access-vcqzj\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033427 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-multus-conf-dir\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b773f2a8-5f9d-4a6f-850f-3c37ae6f4705-serviceca\") pod \"node-ca-clkch\" (UID: \"b773f2a8-5f9d-4a6f-850f-3c37ae6f4705\") " pod="openshift-image-registry/node-ca-clkch" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033504 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4zm\" (UniqueName: \"kubernetes.io/projected/b773f2a8-5f9d-4a6f-850f-3c37ae6f4705-kube-api-access-wc4zm\") pod \"node-ca-clkch\" (UID: \"b773f2a8-5f9d-4a6f-850f-3c37ae6f4705\") " pod="openshift-image-registry/node-ca-clkch" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033529 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-var-lib-openvswitch\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033552 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-var-lib-cni-bin\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033578 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmr4\" (UniqueName: \"kubernetes.io/projected/169267a0-c3db-4d30-b1ab-1e70511f19c6-kube-api-access-zlmr4\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033606 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46bf0532-232f-423a-add2-7c0ff0a5a9a9-kubelet-config\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-ovn-node-metrics-cert\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033676 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-device-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033693 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-sys-fs\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033708 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-kubernetes\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033722 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-var-lib-kubelet\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033741 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-run-ovn\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033762 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-env-overrides\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033786 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-run-multus-certs\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.035647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-sys\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033831 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-tuned\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-multus-socket-dir-parent\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033925 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-var-lib-kubelet\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033949 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-ovnkube-script-lib\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033955 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwww5\" (UniqueName: \"kubernetes.io/projected/b7924db2-6a1f-4ea2-b40c-d04da1a39665-kube-api-access-nwww5\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.033981 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b773f2a8-5f9d-4a6f-850f-3c37ae6f4705-host\") pod \"node-ca-clkch\" (UID: \"b773f2a8-5f9d-4a6f-850f-3c37ae6f4705\") " pod="openshift-image-registry/node-ca-clkch" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034019 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-run-netns\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034051 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-run-systemd\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034058 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-var-lib-openvswitch\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034135 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-run-systemd\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034184 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-run-netns\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034194 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b773f2a8-5f9d-4a6f-850f-3c37ae6f4705-host\") pod \"node-ca-clkch\" (UID: \"b773f2a8-5f9d-4a6f-850f-3c37ae6f4705\") " pod="openshift-image-registry/node-ca-clkch" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034219 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-run-ovn\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034234 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034264 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-system-cni-dir\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034269 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.036343 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034295 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b412e307-f18b-4b6c-87c8-85cff313dece-multus-daemon-config\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034324 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmhd\" (UniqueName: \"kubernetes.io/projected/b412e307-f18b-4b6c-87c8-85cff313dece-kube-api-access-lvmhd\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034378 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46bf0532-232f-423a-add2-7c0ff0a5a9a9-kubelet-config\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034422 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-socket-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034481 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-env-overrides\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034498 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e-agent-certs\") pod \"konnectivity-agent-ltxkf\" (UID: \"c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e\") " pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034534 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2506566-130f-4ac1-9be0-b67224951c9d-tmp-dir\") pod \"node-resolver-z2nhq\" (UID: \"c2506566-130f-4ac1-9be0-b67224951c9d\") " pod="openshift-dns/node-resolver-z2nhq" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034564 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-run-netns\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034584 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b773f2a8-5f9d-4a6f-850f-3c37ae6f4705-serviceca\") pod \"node-ca-clkch\" (UID: \"b773f2a8-5f9d-4a6f-850f-3c37ae6f4705\") " pod="openshift-image-registry/node-ca-clkch" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034588 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-hostroot\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034631 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqbbz\" (UniqueName: \"kubernetes.io/projected/4e2a92d6-a5fb-4091-802d-e1a942c45416-kube-api-access-jqbbz\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034656 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcpbr\" (UniqueName: \"kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr\") pod \"network-check-target-sgwg2\" (UID: \"34296475-0929-4b62-84ce-1666247486b3\") " pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-ovnkube-config\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034701 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2506566-130f-4ac1-9be0-b67224951c9d-hosts-file\") pod \"node-resolver-z2nhq\" (UID: \"c2506566-130f-4ac1-9be0-b67224951c9d\") " pod="openshift-dns/node-resolver-z2nhq" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034786 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034794 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-etc-kubernetes\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034827 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-registration-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.037531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034863 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/195cb462-b600-46bc-ab74-21a87101050a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034893 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e-konnectivity-ca\") pod \"konnectivity-agent-ltxkf\" (UID: \"c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e\") " pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034941 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-sysctl-d\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034964 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-host\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034973 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.034987 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/195cb462-b600-46bc-ab74-21a87101050a-cni-binary-copy\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035011 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrtlp\" (UniqueName: \"kubernetes.io/projected/8e9259fd-d4c3-49ad-8d0b-4cc695654363-kube-api-access-rrtlp\") pod \"iptables-alerter-9drwx\" (UID: \"8e9259fd-d4c3-49ad-8d0b-4cc695654363\") " pod="openshift-network-operator/iptables-alerter-9drwx" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035111 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-kubelet\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035145 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-multus-cni-dir\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035168 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-cnibin\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035193 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-run-k8s-cni-cncf-io\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035230 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035244 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-ovnkube-config\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035252 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-sysconfig\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035277 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-systemd\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.038314 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035303 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e9259fd-d4c3-49ad-8d0b-4cc695654363-iptables-alerter-script\") pod \"iptables-alerter-9drwx\" (UID: \"8e9259fd-d4c3-49ad-8d0b-4cc695654363\") " pod="openshift-network-operator/iptables-alerter-9drwx" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035323 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-kubelet\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035329 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-systemd-units\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-etc-openvswitch\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035378 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-node-log\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035403 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msf64\" (UniqueName: \"kubernetes.io/projected/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-kube-api-access-msf64\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035420 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-systemd-units\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035428 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svj7d\" (UniqueName: \"kubernetes.io/projected/c2506566-130f-4ac1-9be0-b67224951c9d-kube-api-access-svj7d\") pod \"node-resolver-z2nhq\" (UID: \"c2506566-130f-4ac1-9be0-b67224951c9d\") " pod="openshift-dns/node-resolver-z2nhq" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035465 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-var-lib-cni-multus\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035484 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-node-log\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035489 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-modprobe-d\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035494 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e-konnectivity-ca\") pod \"konnectivity-agent-ltxkf\" (UID: \"c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e\") " pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035513 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-run\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035529 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-etc-openvswitch\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035573 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-cni-netd\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035622 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/195cb462-b600-46bc-ab74-21a87101050a-cnibin\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035651 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/195cb462-b600-46bc-ab74-21a87101050a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.039086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035664 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-host-cni-netd\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.039902 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035687 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:23.039902 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035712 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-sysctl-conf\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.039902 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:23.039902 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.035776 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b412e307-f18b-4b6c-87c8-85cff313dece-cni-binary-copy\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.039902 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.035910 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:23.039902 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.036029 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret podName:46bf0532-232f-423a-add2-7c0ff0a5a9a9 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:23.535999658 +0000 UTC m=+3.039832719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret") pod "global-pull-secret-syncer-mh4x5" (UID: "46bf0532-232f-423a-add2-7c0ff0a5a9a9") : object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:23.039902 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.039125 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-ovn-node-metrics-cert\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.039902 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.039248 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e-agent-certs\") pod \"konnectivity-agent-ltxkf\" (UID: \"c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e\") " pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:23.039902 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.039709 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-213.ec2.internal" event={"ID":"ee62c14ad64bc1c5c73f7816f7b32bcc","Type":"ContainerStarted","Data":"a5e2eb3af8f838333a974ba945bdeceedd401a1728b2e41870391b4b6cf634a7"} Sep 10 04:41:23.041276 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.041211 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" event={"ID":"2f78df6d50d8bd4ba0dec82fb21a37c8","Type":"ContainerStarted","Data":"3a476d355567c2c77ed47c202b84cb54030fa51e1ea09ca18246da18f3c43179"} Sep 10 04:41:23.047338 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.047313 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4zm\" (UniqueName: \"kubernetes.io/projected/b773f2a8-5f9d-4a6f-850f-3c37ae6f4705-kube-api-access-wc4zm\") pod \"node-ca-clkch\" (UID: \"b773f2a8-5f9d-4a6f-850f-3c37ae6f4705\") " pod="openshift-image-registry/node-ca-clkch" Sep 10 04:41:23.049311 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.049287 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msf64\" (UniqueName: \"kubernetes.io/projected/a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe-kube-api-access-msf64\") pod \"ovnkube-node-q7lb4\" (UID: \"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.136135 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136093 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svj7d\" (UniqueName: \"kubernetes.io/projected/c2506566-130f-4ac1-9be0-b67224951c9d-kube-api-access-svj7d\") pod \"node-resolver-z2nhq\" (UID: \"c2506566-130f-4ac1-9be0-b67224951c9d\") " pod="openshift-dns/node-resolver-z2nhq" Sep 10 04:41:23.136135 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136140 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-var-lib-cni-multus\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.136365 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136165 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-modprobe-d\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.136365 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136189 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-run\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.136365 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/195cb462-b600-46bc-ab74-21a87101050a-cnibin\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.136365 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136237 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/195cb462-b600-46bc-ab74-21a87101050a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.136365 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136263 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:23.136365 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136285 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-sysctl-conf\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.136365 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136319 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b412e307-f18b-4b6c-87c8-85cff313dece-cni-binary-copy\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.136365 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-lib-modules\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.136365 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136365 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/195cb462-b600-46bc-ab74-21a87101050a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136389 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e9259fd-d4c3-49ad-8d0b-4cc695654363-host-slash\") pod \"iptables-alerter-9drwx\" (UID: \"8e9259fd-d4c3-49ad-8d0b-4cc695654363\") " pod="openshift-network-operator/iptables-alerter-9drwx" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136414 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e2a92d6-a5fb-4091-802d-e1a942c45416-tmp\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/195cb462-b600-46bc-ab74-21a87101050a-system-cni-dir\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136478 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcqzj\" (UniqueName: \"kubernetes.io/projected/195cb462-b600-46bc-ab74-21a87101050a-kube-api-access-vcqzj\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136511 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-multus-conf-dir\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-var-lib-cni-bin\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136561 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmr4\" (UniqueName: \"kubernetes.io/projected/169267a0-c3db-4d30-b1ab-1e70511f19c6-kube-api-access-zlmr4\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136587 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136609 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-device-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136633 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-sys-fs\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136655 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-kubernetes\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136678 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-var-lib-kubelet\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-run-multus-certs\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136725 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-sys\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136746 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-tuned\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136771 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-multus-socket-dir-parent\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.136862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136809 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-var-lib-kubelet\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136834 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwww5\" (UniqueName: \"kubernetes.io/projected/b7924db2-6a1f-4ea2-b40c-d04da1a39665-kube-api-access-nwww5\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-system-cni-dir\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136892 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b412e307-f18b-4b6c-87c8-85cff313dece-multus-daemon-config\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmhd\" (UniqueName: \"kubernetes.io/projected/b412e307-f18b-4b6c-87c8-85cff313dece-kube-api-access-lvmhd\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136938 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-socket-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2506566-130f-4ac1-9be0-b67224951c9d-tmp-dir\") pod \"node-resolver-z2nhq\" (UID: \"c2506566-130f-4ac1-9be0-b67224951c9d\") " pod="openshift-dns/node-resolver-z2nhq" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.136984 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-run-netns\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137009 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-hostroot\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137031 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbbz\" (UniqueName: \"kubernetes.io/projected/4e2a92d6-a5fb-4091-802d-e1a942c45416-kube-api-access-jqbbz\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137047 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpbr\" (UniqueName: \"kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr\") pod \"network-check-target-sgwg2\" (UID: \"34296475-0929-4b62-84ce-1666247486b3\") " pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137068 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2506566-130f-4ac1-9be0-b67224951c9d-hosts-file\") pod \"node-resolver-z2nhq\" (UID: \"c2506566-130f-4ac1-9be0-b67224951c9d\") " pod="openshift-dns/node-resolver-z2nhq" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137090 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-etc-kubernetes\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137106 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-registration-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137121 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/195cb462-b600-46bc-ab74-21a87101050a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137147 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-sysctl-d\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137177 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-host\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.137707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137206 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/195cb462-b600-46bc-ab74-21a87101050a-cni-binary-copy\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137217 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-var-lib-kubelet\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137229 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrtlp\" (UniqueName: \"kubernetes.io/projected/8e9259fd-d4c3-49ad-8d0b-4cc695654363-kube-api-access-rrtlp\") pod \"iptables-alerter-9drwx\" (UID: \"8e9259fd-d4c3-49ad-8d0b-4cc695654363\") " pod="openshift-network-operator/iptables-alerter-9drwx" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-multus-cni-dir\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137306 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-cnibin\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137313 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-socket-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137334 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-run-k8s-cni-cncf-io\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137368 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137392 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-sysconfig\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137418 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-systemd\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137442 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e9259fd-d4c3-49ad-8d0b-4cc695654363-iptables-alerter-script\") pod \"iptables-alerter-9drwx\" (UID: \"8e9259fd-d4c3-49ad-8d0b-4cc695654363\") " pod="openshift-network-operator/iptables-alerter-9drwx" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137520 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-hostroot\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137592 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2506566-130f-4ac1-9be0-b67224951c9d-tmp-dir\") pod \"node-resolver-z2nhq\" (UID: \"c2506566-130f-4ac1-9be0-b67224951c9d\") " pod="openshift-dns/node-resolver-z2nhq" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137675 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-var-lib-kubelet\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137726 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/195cb462-b600-46bc-ab74-21a87101050a-system-cni-dir\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137765 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b412e307-f18b-4b6c-87c8-85cff313dece-multus-daemon-config\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137813 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-run-netns\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137900 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-multus-conf-dir\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.138539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.137943 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-var-lib-cni-bin\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138014 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-system-cni-dir\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138022 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2506566-130f-4ac1-9be0-b67224951c9d-hosts-file\") pod \"node-resolver-z2nhq\" (UID: \"c2506566-130f-4ac1-9be0-b67224951c9d\") " pod="openshift-dns/node-resolver-z2nhq" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138028 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e9259fd-d4c3-49ad-8d0b-4cc695654363-iptables-alerter-script\") pod \"iptables-alerter-9drwx\" (UID: \"8e9259fd-d4c3-49ad-8d0b-4cc695654363\") " pod="openshift-network-operator/iptables-alerter-9drwx" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138080 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-multus-cni-dir\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138097 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-etc-kubernetes\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138123 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-run-multus-certs\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138147 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-registration-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138151 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138170 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-sys\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138192 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-run-k8s-cni-cncf-io\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138200 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-cnibin\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138208 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/195cb462-b600-46bc-ab74-21a87101050a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138246 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-systemd\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138250 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-device-dir\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138290 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-sysconfig\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138299 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-sys-fs\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138311 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-sysctl-d\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.139460 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138341 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-kubernetes\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138363 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-host\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138385 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b7924db2-6a1f-4ea2-b40c-d04da1a39665-etc-selinux\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138395 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-multus-socket-dir-parent\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138888 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/195cb462-b600-46bc-ab74-21a87101050a-cni-binary-copy\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.138927 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/195cb462-b600-46bc-ab74-21a87101050a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.138982 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.139026 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b412e307-f18b-4b6c-87c8-85cff313dece-cni-binary-copy\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.139044 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs podName:169267a0-c3db-4d30-b1ab-1e70511f19c6 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:23.639026573 +0000 UTC m=+3.142859616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs") pod "network-metrics-daemon-n2dx5" (UID: "169267a0-c3db-4d30-b1ab-1e70511f19c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.139182 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-sysctl-conf\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.139233 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-run\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.139304 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-modprobe-d\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.139339 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/195cb462-b600-46bc-ab74-21a87101050a-cnibin\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.139399 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e9259fd-d4c3-49ad-8d0b-4cc695654363-host-slash\") pod \"iptables-alerter-9drwx\" (UID: \"8e9259fd-d4c3-49ad-8d0b-4cc695654363\") " pod="openshift-network-operator/iptables-alerter-9drwx" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.139504 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e2a92d6-a5fb-4091-802d-e1a942c45416-lib-modules\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.139700 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/195cb462-b600-46bc-ab74-21a87101050a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.140308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.139937 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e2a92d6-a5fb-4091-802d-e1a942c45416-tmp\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.140888 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.140325 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b412e307-f18b-4b6c-87c8-85cff313dece-host-var-lib-cni-multus\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.141135 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.141093 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e2a92d6-a5fb-4091-802d-e1a942c45416-etc-tuned\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.150289 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.150261 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 04:41:23.150289 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.150291 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 04:41:23.150488 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.150305 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fcpbr for pod openshift-network-diagnostics/network-check-target-sgwg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:23.150488 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.150376 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr podName:34296475-0929-4b62-84ce-1666247486b3 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:23.650358115 +0000 UTC m=+3.154191154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fcpbr" (UniqueName: "kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr") pod "network-check-target-sgwg2" (UID: "34296475-0929-4b62-84ce-1666247486b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:23.152593 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.152497 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwww5\" (UniqueName: \"kubernetes.io/projected/b7924db2-6a1f-4ea2-b40c-d04da1a39665-kube-api-access-nwww5\") pod \"aws-ebs-csi-driver-node-f5lhz\" (UID: \"b7924db2-6a1f-4ea2-b40c-d04da1a39665\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.152593 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.152546 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrtlp\" (UniqueName: \"kubernetes.io/projected/8e9259fd-d4c3-49ad-8d0b-4cc695654363-kube-api-access-rrtlp\") pod \"iptables-alerter-9drwx\" (UID: \"8e9259fd-d4c3-49ad-8d0b-4cc695654363\") " pod="openshift-network-operator/iptables-alerter-9drwx" Sep 10 04:41:23.153120 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.153096 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmr4\" (UniqueName: \"kubernetes.io/projected/169267a0-c3db-4d30-b1ab-1e70511f19c6-kube-api-access-zlmr4\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:23.153647 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.153622 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svj7d\" (UniqueName: \"kubernetes.io/projected/c2506566-130f-4ac1-9be0-b67224951c9d-kube-api-access-svj7d\") pod \"node-resolver-z2nhq\" (UID: \"c2506566-130f-4ac1-9be0-b67224951c9d\") " pod="openshift-dns/node-resolver-z2nhq" Sep 10 04:41:23.153767 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.153748 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmhd\" (UniqueName: \"kubernetes.io/projected/b412e307-f18b-4b6c-87c8-85cff313dece-kube-api-access-lvmhd\") pod \"multus-grjvl\" (UID: \"b412e307-f18b-4b6c-87c8-85cff313dece\") " pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.154295 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.154273 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcqzj\" (UniqueName: \"kubernetes.io/projected/195cb462-b600-46bc-ab74-21a87101050a-kube-api-access-vcqzj\") pod \"multus-additional-cni-plugins-sg5jd\" (UID: \"195cb462-b600-46bc-ab74-21a87101050a\") " pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.155375 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.155355 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbbz\" (UniqueName: \"kubernetes.io/projected/4e2a92d6-a5fb-4091-802d-e1a942c45416-kube-api-access-jqbbz\") pod \"tuned-wkj57\" (UID: \"4e2a92d6-a5fb-4091-802d-e1a942c45416\") " pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.228500 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.228465 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-clkch" Sep 10 04:41:23.238480 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.238439 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:23.250243 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.250215 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:23.256065 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.256033 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wkj57" Sep 10 04:41:23.263786 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.263761 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z2nhq" Sep 10 04:41:23.273587 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.273535 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sg5jd" Sep 10 04:41:23.281391 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.281371 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9drwx" Sep 10 04:41:23.288989 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.288971 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-grjvl" Sep 10 04:41:23.295473 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.295438 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" Sep 10 04:41:23.541303 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.541204 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:23.541505 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.541386 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:23.541505 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.541466 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret podName:46bf0532-232f-423a-add2-7c0ff0a5a9a9 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:24.541435792 +0000 UTC m=+4.045268835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret") pod "global-pull-secret-syncer-mh4x5" (UID: "46bf0532-232f-423a-add2-7c0ff0a5a9a9") : object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:23.642329 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.642280 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:23.642511 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.642393 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:23.642511 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.642485 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs podName:169267a0-c3db-4d30-b1ab-1e70511f19c6 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:24.642465725 +0000 UTC m=+4.146298771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs") pod "network-metrics-daemon-n2dx5" (UID: "169267a0-c3db-4d30-b1ab-1e70511f19c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:23.743492 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.743437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpbr\" (UniqueName: \"kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr\") pod \"network-check-target-sgwg2\" (UID: \"34296475-0929-4b62-84ce-1666247486b3\") " pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:23.743692 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.743638 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 04:41:23.743692 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.743663 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 04:41:23.743692 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.743674 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fcpbr for pod openshift-network-diagnostics/network-check-target-sgwg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:23.743854 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:23.743736 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr podName:34296475-0929-4b62-84ce-1666247486b3 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:24.743720959 +0000 UTC m=+4.247554017 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcpbr" (UniqueName: "kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr") pod "network-check-target-sgwg2" (UID: "34296475-0929-4b62-84ce-1666247486b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:23.969880 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.969803 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 04:36:22 +0000 UTC" deadline="2028-02-19 22:50:35.192874228 +0000 UTC" Sep 10 04:41:23.969880 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:23.969836 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12666h9m11.223041247s" Sep 10 04:41:23.982401 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:23.982187 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c8dbec_7b2c_4c5a_a06a_1c1b43e775fe.slice/crio-98d3ce20e68fc2313572d6d94c9b8adb78d8f032770214f2af65f566418868d3 WatchSource:0}: Error finding container 98d3ce20e68fc2313572d6d94c9b8adb78d8f032770214f2af65f566418868d3: Status 404 returned error can't find the container with id 98d3ce20e68fc2313572d6d94c9b8adb78d8f032770214f2af65f566418868d3 Sep 10 04:41:23.988233 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:23.988204 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9259fd_d4c3_49ad_8d0b_4cc695654363.slice/crio-f3726b644944efa4acae1ef69beea81f589f1c78426a5cc449d665ef263e32d3 WatchSource:0}: Error finding container f3726b644944efa4acae1ef69beea81f589f1c78426a5cc449d665ef263e32d3: Status 404 returned error can't find the container with id f3726b644944efa4acae1ef69beea81f589f1c78426a5cc449d665ef263e32d3 Sep 10 04:41:23.989132 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:23.989103 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2506566_130f_4ac1_9be0_b67224951c9d.slice/crio-4fc3b9da4a612d5b5f2da1041bae666e9c0490c56cf7ecc8dbcf4feb331757f6 WatchSource:0}: Error finding container 4fc3b9da4a612d5b5f2da1041bae666e9c0490c56cf7ecc8dbcf4feb331757f6: Status 404 returned error can't find the container with id 4fc3b9da4a612d5b5f2da1041bae666e9c0490c56cf7ecc8dbcf4feb331757f6 Sep 10 04:41:23.990527 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:23.990498 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb412e307_f18b_4b6c_87c8_85cff313dece.slice/crio-6a5f860ddc6a95d5d54b862fcc615740ee2d080d27ff43cdce49a489ea82846e WatchSource:0}: Error finding container 6a5f860ddc6a95d5d54b862fcc615740ee2d080d27ff43cdce49a489ea82846e: Status 404 returned error can't find the container with id 6a5f860ddc6a95d5d54b862fcc615740ee2d080d27ff43cdce49a489ea82846e Sep 10 04:41:23.991422 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:23.991386 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195cb462_b600_46bc_ab74_21a87101050a.slice/crio-88400c3c1a947c8cf1a09965c272561eafeb1947ac36b75af057ebd36c1b84dd WatchSource:0}: Error finding container 88400c3c1a947c8cf1a09965c272561eafeb1947ac36b75af057ebd36c1b84dd: Status 404 returned error can't find the container with id 88400c3c1a947c8cf1a09965c272561eafeb1947ac36b75af057ebd36c1b84dd Sep 10 04:41:23.992556 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:23.992537 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2a92d6_a5fb_4091_802d_e1a942c45416.slice/crio-0b67c3544b2097f788c70424319350159ec4200ea3d26f91ea2ec96ea4f0f4b3 WatchSource:0}: Error finding container 0b67c3544b2097f788c70424319350159ec4200ea3d26f91ea2ec96ea4f0f4b3: Status 404 returned error can't find the container with id 0b67c3544b2097f788c70424319350159ec4200ea3d26f91ea2ec96ea4f0f4b3 Sep 10 04:41:23.994820 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:23.994508 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb773f2a8_5f9d_4a6f_850f_3c37ae6f4705.slice/crio-118524dd5f90c43f033329b2d05fc5418262044d3b9aa77d0f49ec90aa69695f WatchSource:0}: Error finding container 118524dd5f90c43f033329b2d05fc5418262044d3b9aa77d0f49ec90aa69695f: Status 404 returned error can't find the container with id 118524dd5f90c43f033329b2d05fc5418262044d3b9aa77d0f49ec90aa69695f Sep 10 04:41:23.995398 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:23.995254 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6759e6f_cc18_4a5e_bdbf_48aa0bdca13e.slice/crio-b49967bf137894608f734aea1a2096b6876d38ee08cb4d9cfb5b98e8513a2563 WatchSource:0}: Error finding container b49967bf137894608f734aea1a2096b6876d38ee08cb4d9cfb5b98e8513a2563: Status 404 returned error can't find the container with id b49967bf137894608f734aea1a2096b6876d38ee08cb4d9cfb5b98e8513a2563 Sep 10 04:41:23.996967 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:41:23.996939 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7924db2_6a1f_4ea2_b40c_d04da1a39665.slice/crio-2341a27e53c6d18809ddc0964877dcc6275146161bce9d4e5a63977fb04c334f WatchSource:0}: Error finding container 2341a27e53c6d18809ddc0964877dcc6275146161bce9d4e5a63977fb04c334f: Status 404 returned error can't find the container with id 2341a27e53c6d18809ddc0964877dcc6275146161bce9d4e5a63977fb04c334f Sep 10 04:41:24.030196 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.030177 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:24.030287 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:24.030271 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:24.044101 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.044075 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wkj57" event={"ID":"4e2a92d6-a5fb-4091-802d-e1a942c45416","Type":"ContainerStarted","Data":"0b67c3544b2097f788c70424319350159ec4200ea3d26f91ea2ec96ea4f0f4b3"} Sep 10 04:41:24.045527 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.045499 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg5jd" event={"ID":"195cb462-b600-46bc-ab74-21a87101050a","Type":"ContainerStarted","Data":"88400c3c1a947c8cf1a09965c272561eafeb1947ac36b75af057ebd36c1b84dd"} Sep 10 04:41:24.046668 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.046646 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z2nhq" event={"ID":"c2506566-130f-4ac1-9be0-b67224951c9d","Type":"ContainerStarted","Data":"4fc3b9da4a612d5b5f2da1041bae666e9c0490c56cf7ecc8dbcf4feb331757f6"} Sep 10 04:41:24.047806 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.047781 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ltxkf" event={"ID":"c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e","Type":"ContainerStarted","Data":"b49967bf137894608f734aea1a2096b6876d38ee08cb4d9cfb5b98e8513a2563"} Sep 10 04:41:24.048990 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.048968 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-grjvl" event={"ID":"b412e307-f18b-4b6c-87c8-85cff313dece","Type":"ContainerStarted","Data":"6a5f860ddc6a95d5d54b862fcc615740ee2d080d27ff43cdce49a489ea82846e"} Sep 10 04:41:24.051654 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.051628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9drwx" event={"ID":"8e9259fd-d4c3-49ad-8d0b-4cc695654363","Type":"ContainerStarted","Data":"f3726b644944efa4acae1ef69beea81f589f1c78426a5cc449d665ef263e32d3"} Sep 10 04:41:24.056159 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.056129 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" event={"ID":"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe","Type":"ContainerStarted","Data":"98d3ce20e68fc2313572d6d94c9b8adb78d8f032770214f2af65f566418868d3"} Sep 10 04:41:24.059080 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.059057 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" event={"ID":"b7924db2-6a1f-4ea2-b40c-d04da1a39665","Type":"ContainerStarted","Data":"2341a27e53c6d18809ddc0964877dcc6275146161bce9d4e5a63977fb04c334f"} Sep 10 04:41:24.062655 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.062632 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-clkch" event={"ID":"b773f2a8-5f9d-4a6f-850f-3c37ae6f4705","Type":"ContainerStarted","Data":"118524dd5f90c43f033329b2d05fc5418262044d3b9aa77d0f49ec90aa69695f"} Sep 10 04:41:24.549608 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.549516 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:24.549778 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:24.549719 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:24.549843 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:24.549783 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret podName:46bf0532-232f-423a-add2-7c0ff0a5a9a9 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:26.549765502 +0000 UTC m=+6.053598561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret") pod "global-pull-secret-syncer-mh4x5" (UID: "46bf0532-232f-423a-add2-7c0ff0a5a9a9") : object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:24.650944 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.650904 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:24.651127 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:24.651082 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:24.651214 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:24.651146 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs podName:169267a0-c3db-4d30-b1ab-1e70511f19c6 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:26.651128008 +0000 UTC m=+6.154961047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs") pod "network-metrics-daemon-n2dx5" (UID: "169267a0-c3db-4d30-b1ab-1e70511f19c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:24.752066 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.751997 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpbr\" (UniqueName: \"kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr\") pod \"network-check-target-sgwg2\" (UID: \"34296475-0929-4b62-84ce-1666247486b3\") " pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:24.752298 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:24.752203 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 04:41:24.752298 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:24.752226 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 04:41:24.752298 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:24.752240 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fcpbr for pod openshift-network-diagnostics/network-check-target-sgwg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:24.752472 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:24.752303 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr podName:34296475-0929-4b62-84ce-1666247486b3 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:26.752282035 +0000 UTC m=+6.256115080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcpbr" (UniqueName: "kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr") pod "network-check-target-sgwg2" (UID: "34296475-0929-4b62-84ce-1666247486b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:24.970858 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.970612 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 04:36:22 +0000 UTC" deadline="2028-06-18 11:11:56.853213246 +0000 UTC" Sep 10 04:41:24.970858 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:24.970673 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15534h30m31.882543889s" Sep 10 04:41:25.032041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:25.031989 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:25.032218 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:25.032140 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:25.032656 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:25.032634 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:25.032761 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:25.032735 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:25.079615 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:25.079574 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-213.ec2.internal" event={"ID":"ee62c14ad64bc1c5c73f7816f7b32bcc","Type":"ContainerStarted","Data":"3010696b78f6b5f34188dd11d60c903dde9fb9708b821e5a205a6a5d6ca6743c"} Sep 10 04:41:25.097962 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:25.097924 2572 generic.go:356] "Generic (PLEG): container finished" podID="2f78df6d50d8bd4ba0dec82fb21a37c8" containerID="33fa34f59b22f2f6060c896219b727b2ee0b5445c655e30801aff29981b87ec6" exitCode=0 Sep 10 04:41:25.098149 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:25.097973 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" event={"ID":"2f78df6d50d8bd4ba0dec82fb21a37c8","Type":"ContainerDied","Data":"33fa34f59b22f2f6060c896219b727b2ee0b5445c655e30801aff29981b87ec6"} Sep 10 04:41:25.112937 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:25.112873 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-213.ec2.internal" podStartSLOduration=3.112851855 podStartE2EDuration="3.112851855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:25.095649892 +0000 UTC m=+4.599482953" watchObservedRunningTime="2026-09-10 04:41:25.112851855 +0000 UTC m=+4.616684918" Sep 10 04:41:26.031174 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:26.031086 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:26.031881 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:26.031230 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:26.124541 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:26.124505 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" event={"ID":"2f78df6d50d8bd4ba0dec82fb21a37c8","Type":"ContainerStarted","Data":"03580b030afd2912d98753460907d9e2ecad370846507b6343a838b3f3987ad2"} Sep 10 04:41:26.565093 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:26.565052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:26.565272 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:26.565245 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:26.565336 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:26.565309 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret podName:46bf0532-232f-423a-add2-7c0ff0a5a9a9 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:30.56529074 +0000 UTC m=+10.069123781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret") pod "global-pull-secret-syncer-mh4x5" (UID: "46bf0532-232f-423a-add2-7c0ff0a5a9a9") : object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:26.665853 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:26.665813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:26.666088 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:26.665993 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:26.666088 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:26.666072 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs podName:169267a0-c3db-4d30-b1ab-1e70511f19c6 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:30.66605325 +0000 UTC m=+10.169886292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs") pod "network-metrics-daemon-n2dx5" (UID: "169267a0-c3db-4d30-b1ab-1e70511f19c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:26.767192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:26.767152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpbr\" (UniqueName: \"kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr\") pod \"network-check-target-sgwg2\" (UID: \"34296475-0929-4b62-84ce-1666247486b3\") " pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:26.767406 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:26.767360 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 04:41:26.767406 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:26.767384 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 04:41:26.767406 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:26.767396 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fcpbr for pod openshift-network-diagnostics/network-check-target-sgwg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:26.767621 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:26.767471 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr podName:34296475-0929-4b62-84ce-1666247486b3 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:30.767439288 +0000 UTC m=+10.271272330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcpbr" (UniqueName: "kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr") pod "network-check-target-sgwg2" (UID: "34296475-0929-4b62-84ce-1666247486b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:27.033511 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:27.033425 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:27.033941 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:27.033581 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:27.033997 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:27.033963 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:27.034086 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:27.034065 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:28.031079 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:28.031039 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:28.031273 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:28.031182 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:29.031407 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:29.030926 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:29.031407 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:29.031055 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:29.032019 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:29.031989 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:29.032136 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:29.032114 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:30.030536 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:30.030428 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:30.030702 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:30.030591 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:30.603703 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:30.603665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:30.604211 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:30.603780 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:30.604211 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:30.603853 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret podName:46bf0532-232f-423a-add2-7c0ff0a5a9a9 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:38.603834096 +0000 UTC m=+18.107667139 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret") pod "global-pull-secret-syncer-mh4x5" (UID: "46bf0532-232f-423a-add2-7c0ff0a5a9a9") : object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:30.704693 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:30.704655 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:30.704907 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:30.704876 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:30.705044 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:30.704994 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs podName:169267a0-c3db-4d30-b1ab-1e70511f19c6 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:38.704954499 +0000 UTC m=+18.208787555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs") pod "network-metrics-daemon-n2dx5" (UID: "169267a0-c3db-4d30-b1ab-1e70511f19c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:30.806111 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:30.806064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpbr\" (UniqueName: \"kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr\") pod \"network-check-target-sgwg2\" (UID: \"34296475-0929-4b62-84ce-1666247486b3\") " pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:30.806307 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:30.806270 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 04:41:30.806307 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:30.806295 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 04:41:30.806407 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:30.806308 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fcpbr for pod openshift-network-diagnostics/network-check-target-sgwg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:30.806407 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:30.806371 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr podName:34296475-0929-4b62-84ce-1666247486b3 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:38.806355835 +0000 UTC m=+18.310188893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcpbr" (UniqueName: "kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr") pod "network-check-target-sgwg2" (UID: "34296475-0929-4b62-84ce-1666247486b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:31.031816 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:31.031741 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:31.031816 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:31.031780 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:31.032008 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:31.031875 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:31.032008 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:31.031979 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:32.030534 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:32.030504 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:32.030921 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:32.030635 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:33.030379 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:33.030340 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:33.030563 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:33.030497 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:33.030563 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:33.030546 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:33.030972 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:33.030693 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:34.030488 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:34.030431 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:34.030682 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:34.030612 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:35.030432 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:35.030388 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:35.030613 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:35.030442 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:35.030613 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:35.030564 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:35.031027 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:35.030699 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:36.031361 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:36.031159 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:36.031803 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:36.031468 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:37.031236 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:37.031203 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:37.031425 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:37.031205 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:37.031425 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:37.031344 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:37.031857 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:37.031429 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:38.031020 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:38.030914 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:38.031201 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:38.031027 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:38.668321 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:38.668281 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:38.668789 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:38.668424 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:38.668789 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:38.668503 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret podName:46bf0532-232f-423a-add2-7c0ff0a5a9a9 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:54.668489235 +0000 UTC m=+34.172322274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret") pod "global-pull-secret-syncer-mh4x5" (UID: "46bf0532-232f-423a-add2-7c0ff0a5a9a9") : object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:38.769005 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:38.768963 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:38.769184 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:38.769099 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:38.769184 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:38.769168 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs podName:169267a0-c3db-4d30-b1ab-1e70511f19c6 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:54.769149859 +0000 UTC m=+34.272982913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs") pod "network-metrics-daemon-n2dx5" (UID: "169267a0-c3db-4d30-b1ab-1e70511f19c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:38.870255 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:38.870219 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpbr\" (UniqueName: \"kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr\") pod \"network-check-target-sgwg2\" (UID: \"34296475-0929-4b62-84ce-1666247486b3\") " pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:38.870440 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:38.870416 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 04:41:38.870521 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:38.870463 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 04:41:38.870521 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:38.870478 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fcpbr for pod openshift-network-diagnostics/network-check-target-sgwg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:38.870633 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:38.870538 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr podName:34296475-0929-4b62-84ce-1666247486b3 nodeName:}" failed. No retries permitted until 2026-09-10 04:41:54.870522808 +0000 UTC m=+34.374355847 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcpbr" (UniqueName: "kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr") pod "network-check-target-sgwg2" (UID: "34296475-0929-4b62-84ce-1666247486b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:39.030540 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:39.030503 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:39.030739 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:39.030503 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:39.030739 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:39.030647 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:39.030846 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:39.030737 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:40.030532 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:40.030494 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:40.030960 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:40.030616 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:41.031340 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:41.031304 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:41.031822 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:41.031440 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:41.031822 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:41.031484 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:41.031822 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:41.031602 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:42.033667 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.033474 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:42.033667 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:42.033607 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:42.159707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.159395 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-grjvl" event={"ID":"b412e307-f18b-4b6c-87c8-85cff313dece","Type":"ContainerStarted","Data":"2fb3d7227179f3cb577740e8c1f3ef87221a589ee183b194eddede5771bec88f"} Sep 10 04:41:42.161898 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.161795 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" event={"ID":"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe","Type":"ContainerStarted","Data":"37a766c9021d649e54def1ec0a9ab3138a54341bdb746d1516773acedf74b4f7"} Sep 10 04:41:42.162046 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.161898 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" event={"ID":"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe","Type":"ContainerStarted","Data":"5f6f86c4e73ba64ac1122925debe67d6dbe946014c464e6684ae0d7076831052"} Sep 10 04:41:42.163412 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.163384 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" event={"ID":"b7924db2-6a1f-4ea2-b40c-d04da1a39665","Type":"ContainerStarted","Data":"350cc6afe81f1fd514ff65b59828b606f1282b16dbe98e6db760fdb95eb6387d"} Sep 10 04:41:42.164976 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.164933 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-clkch" event={"ID":"b773f2a8-5f9d-4a6f-850f-3c37ae6f4705","Type":"ContainerStarted","Data":"da05bbd789a0aca8a40c37e79641c6bc02754c9e41a3bd4aca2157378095cd09"} Sep 10 04:41:42.167008 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.166983 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wkj57" event={"ID":"4e2a92d6-a5fb-4091-802d-e1a942c45416","Type":"ContainerStarted","Data":"68dac868cda41b1d8ffc92b4a1e058e7fcf837d78c8d4873d639d16eae404c72"} Sep 10 04:41:42.169146 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.169075 2572 generic.go:356] "Generic (PLEG): container finished" podID="195cb462-b600-46bc-ab74-21a87101050a" containerID="301280483c0bee57521de27f441868564640e7be18e702390e4690625aa1ad39" exitCode=0 Sep 10 04:41:42.169146 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.169138 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg5jd" event={"ID":"195cb462-b600-46bc-ab74-21a87101050a","Type":"ContainerDied","Data":"301280483c0bee57521de27f441868564640e7be18e702390e4690625aa1ad39"} Sep 10 04:41:42.171115 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.171093 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z2nhq" event={"ID":"c2506566-130f-4ac1-9be0-b67224951c9d","Type":"ContainerStarted","Data":"0f5462d723007f10d049d2f264cd6179055c5386353a5ab88f3ba921041a12cd"} Sep 10 04:41:42.172680 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.172658 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ltxkf" event={"ID":"c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e","Type":"ContainerStarted","Data":"b648768904c44c4168841d34b74a1ea2bdea9446c15f444228385a49fa9dd9b6"} Sep 10 04:41:42.179673 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.179622 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-213.ec2.internal" podStartSLOduration=20.179606193 podStartE2EDuration="20.179606193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:26.141015877 +0000 UTC m=+5.644848939" watchObservedRunningTime="2026-09-10 04:41:42.179606193 +0000 UTC m=+21.683439258" Sep 10 04:41:42.179847 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.179704 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-grjvl" podStartSLOduration=3.253361652 podStartE2EDuration="21.179699282s" totalImagesPullingTime="17.92633763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:42.178925682 +0000 UTC m=+21.682758743" watchObservedRunningTime="2026-09-10 04:41:42.179699282 +0000 UTC m=+21.683532343" Sep 10 04:41:42.196254 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.196203 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ltxkf" podStartSLOduration=7.466628274 podStartE2EDuration="21.196189238s" totalImagesPullingTime="13.729560964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:42.19567454 +0000 UTC m=+21.699507601" watchObservedRunningTime="2026-09-10 04:41:42.196189238 +0000 UTC m=+21.700022299" Sep 10 04:41:42.219352 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.219295 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wkj57" podStartSLOduration=3.363050308 podStartE2EDuration="21.21925153s" totalImagesPullingTime="17.856201222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:42.218878711 +0000 UTC m=+21.722711772" watchObservedRunningTime="2026-09-10 04:41:42.21925153 +0000 UTC m=+21.723084592" Sep 10 04:41:42.234218 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.234170 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-clkch" podStartSLOduration=3.40122839 podStartE2EDuration="21.234148985s" totalImagesPullingTime="17.832920595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:42.233773272 +0000 UTC m=+21.737606333" watchObservedRunningTime="2026-09-10 04:41:42.234148985 +0000 UTC m=+21.737982048" Sep 10 04:41:42.277731 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.277679 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z2nhq" podStartSLOduration=3.417856544 podStartE2EDuration="21.277663105s" totalImagesPullingTime="17.859806561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:42.25130966 +0000 UTC m=+21.755142722" watchObservedRunningTime="2026-09-10 04:41:42.277663105 +0000 UTC m=+21.781496222" Sep 10 04:41:42.900510 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.900472 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:42.901158 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:42.901136 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:43.030689 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:43.030655 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:43.030867 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:43.030658 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:43.030947 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:43.030924 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:43.030993 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:43.030789 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:43.178875 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:43.178842 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" event={"ID":"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe","Type":"ContainerStarted","Data":"42d1ac788e6af941acab9a541556d249443197ff58f632dbcb45fd9390d6168f"} Sep 10 04:41:43.179791 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:43.178915 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" event={"ID":"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe","Type":"ContainerStarted","Data":"f373877659ec85b68f99a7683f573ba53f9066eaec75610868a16ad28c530759"} Sep 10 04:41:43.179791 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:43.178964 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" event={"ID":"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe","Type":"ContainerStarted","Data":"90e2b468e0b99b3022529c2581e9661aa21fce8dfc499bb78df6c0fcdc21761e"} Sep 10 04:41:43.179791 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:43.179035 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" event={"ID":"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe","Type":"ContainerStarted","Data":"be3607b726b8d884329ec83b7799895575a79f2372d2893998650445f7384d52"} Sep 10 04:41:43.179791 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:43.179739 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:43.179925 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:43.179906 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ltxkf" Sep 10 04:41:43.361915 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:43.361882 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 04:41:44.000388 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:44.000267 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T04:41:43.361907246Z","UUID":"3d4998d4-a052-4077-a35d-89bcfda0181b","Handler":null,"Name":"","Endpoint":""} Sep 10 04:41:44.002350 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:44.002324 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 04:41:44.002531 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:44.002363 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 04:41:44.030643 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:44.030607 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:44.030818 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:44.030732 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:44.183432 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:44.183387 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" event={"ID":"b7924db2-6a1f-4ea2-b40c-d04da1a39665","Type":"ContainerStarted","Data":"aa7cc40028156736aa8b9134629ad47fac9ee2776728e648261e857db211f8e2"} Sep 10 04:41:44.185414 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:44.185379 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9drwx" event={"ID":"8e9259fd-d4c3-49ad-8d0b-4cc695654363","Type":"ContainerStarted","Data":"5967655a21ec3911df4ced489ae7ff81a0d2028194ce3ff15c149d28e8d0e2e9"} Sep 10 04:41:44.208502 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:44.208432 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9drwx" podStartSLOduration=5.348519627 podStartE2EDuration="23.208413268s" totalImagesPullingTime="17.859893641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:44.207682491 +0000 UTC m=+23.711515557" watchObservedRunningTime="2026-09-10 04:41:44.208413268 +0000 UTC m=+23.712246329" Sep 10 04:41:45.030527 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:45.030261 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:45.030750 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:45.030261 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:45.030750 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:45.030683 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:45.030864 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:45.030805 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:45.192126 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:45.192086 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" event={"ID":"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe","Type":"ContainerStarted","Data":"73200db78b9f601d38eabd2989dab2848bb6b516ab76f6d8b21db923d160269a"} Sep 10 04:41:45.194562 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:45.194527 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" event={"ID":"b7924db2-6a1f-4ea2-b40c-d04da1a39665","Type":"ContainerStarted","Data":"73f9563b1c6619f76337233df64f6bf3876aae8a3ff9209ed03e25ad92dab5e9"} Sep 10 04:41:45.221322 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:45.221270 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f5lhz" podStartSLOduration=4.195779439 podStartE2EDuration="24.221249631s" totalImagesPullingTime="20.025470192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:45.221127037 +0000 UTC m=+24.724960097" watchObservedRunningTime="2026-09-10 04:41:45.221249631 +0000 UTC m=+24.725082693" Sep 10 04:41:46.031102 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:46.031062 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:46.031336 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:46.031199 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:47.031261 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:47.031173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:47.031701 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:47.031294 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:47.031701 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:47.031325 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:47.031701 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:47.031378 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:47.202475 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:47.202190 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" event={"ID":"a5c8dbec-7b2c-4c5a-a06a-1c1b43e775fe","Type":"ContainerStarted","Data":"41defaf24284e8c0e42b5e202af6e5ee49ec403f428a6b6538b4a62782c9a0d6"} Sep 10 04:41:47.202661 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:47.202530 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:47.202661 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:47.202599 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:47.204193 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:47.204168 2572 generic.go:356] "Generic (PLEG): container finished" podID="195cb462-b600-46bc-ab74-21a87101050a" containerID="e97ab8c272f3656ef0313c404c52c4e61e3bc726b383e7a0450b6f86987dbfb8" exitCode=0 Sep 10 04:41:47.204302 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:47.204203 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg5jd" event={"ID":"195cb462-b600-46bc-ab74-21a87101050a","Type":"ContainerDied","Data":"e97ab8c272f3656ef0313c404c52c4e61e3bc726b383e7a0450b6f86987dbfb8"} Sep 10 04:41:47.218848 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:47.218812 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:47.230889 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:47.230844 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" podStartSLOduration=8.32733971 podStartE2EDuration="26.230829091s" totalImagesPullingTime="17.903489381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:47.229350111 +0000 UTC m=+26.733183169" watchObservedRunningTime="2026-09-10 04:41:47.230829091 +0000 UTC m=+26.734662151" Sep 10 04:41:47.564476 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:47.564439 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z2nhq_c2506566-130f-4ac1-9be0-b67224951c9d/dns-node-resolver/0.log" Sep 10 04:41:48.030261 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.030227 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:48.030479 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:48.030340 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:48.143230 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.143146 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-clkch_b773f2a8-5f9d-4a6f-850f-3c37ae6f4705/node-ca/0.log" Sep 10 04:41:48.211512 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.209504 2572 generic.go:356] "Generic (PLEG): container finished" podID="195cb462-b600-46bc-ab74-21a87101050a" containerID="3cd9473743ae0a7dea62414d9303bf587235f955761c0ae4b3d6d737396804e0" exitCode=0 Sep 10 04:41:48.211512 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.211077 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg5jd" event={"ID":"195cb462-b600-46bc-ab74-21a87101050a","Type":"ContainerDied","Data":"3cd9473743ae0a7dea62414d9303bf587235f955761c0ae4b3d6d737396804e0"} Sep 10 04:41:48.211512 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.211406 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:48.233137 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.233103 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:41:48.799376 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.799330 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mh4x5"] Sep 10 04:41:48.799569 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.799492 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:48.799640 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:48.799608 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:48.802823 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.802793 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n2dx5"] Sep 10 04:41:48.802980 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.802921 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:48.803053 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:48.803029 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:48.814229 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.814151 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sgwg2"] Sep 10 04:41:48.814374 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:48.814265 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:48.814440 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:48.814371 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:49.214125 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:49.214089 2572 generic.go:356] "Generic (PLEG): container finished" podID="195cb462-b600-46bc-ab74-21a87101050a" containerID="7b51d3b23810b7949f57de688d6f261296da9c132e67cf59bd8ab32a1e72b10c" exitCode=0 Sep 10 04:41:49.214797 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:49.214184 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg5jd" event={"ID":"195cb462-b600-46bc-ab74-21a87101050a","Type":"ContainerDied","Data":"7b51d3b23810b7949f57de688d6f261296da9c132e67cf59bd8ab32a1e72b10c"} Sep 10 04:41:50.030852 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:50.030811 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:50.031033 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:50.030811 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:50.031033 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:50.030934 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:50.031150 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:50.031044 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:51.031532 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:51.031278 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:51.031974 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:51.031654 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:52.030605 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:52.030566 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:52.030799 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:52.030695 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:52.030799 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:52.030714 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:52.030894 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:52.030839 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:53.031178 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:53.031144 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:53.031686 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:53.031281 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:54.030636 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:54.030598 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:54.030824 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:54.030598 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:54.030824 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:54.030708 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:54.030824 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:54.030773 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:54.228117 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:54.228083 2572 generic.go:356] "Generic (PLEG): container finished" podID="195cb462-b600-46bc-ab74-21a87101050a" containerID="9c71ae45fa13c3d33afd1c3ea6e8b3b733eaabcbcb24e69fd56d8c220d55bbb2" exitCode=0 Sep 10 04:41:54.228528 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:54.228130 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg5jd" event={"ID":"195cb462-b600-46bc-ab74-21a87101050a","Type":"ContainerDied","Data":"9c71ae45fa13c3d33afd1c3ea6e8b3b733eaabcbcb24e69fd56d8c220d55bbb2"} Sep 10 04:41:54.680788 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:54.680702 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:54.681015 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:54.680817 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:54.681015 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:54.680876 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret podName:46bf0532-232f-423a-add2-7c0ff0a5a9a9 nodeName:}" failed. No retries permitted until 2026-09-10 04:42:26.680861399 +0000 UTC m=+66.184694450 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret") pod "global-pull-secret-syncer-mh4x5" (UID: "46bf0532-232f-423a-add2-7c0ff0a5a9a9") : object "kube-system"/"original-pull-secret" not registered Sep 10 04:41:54.782062 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:54.782029 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:54.782261 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:54.782160 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:54.782261 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:54.782216 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs podName:169267a0-c3db-4d30-b1ab-1e70511f19c6 nodeName:}" failed. No retries permitted until 2026-09-10 04:42:26.782203179 +0000 UTC m=+66.286036218 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs") pod "network-metrics-daemon-n2dx5" (UID: "169267a0-c3db-4d30-b1ab-1e70511f19c6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 04:41:54.882332 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:54.882290 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpbr\" (UniqueName: \"kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr\") pod \"network-check-target-sgwg2\" (UID: \"34296475-0929-4b62-84ce-1666247486b3\") " pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:54.882526 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:54.882418 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 04:41:54.882526 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:54.882435 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 04:41:54.882526 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:54.882462 2572 projected.go:196] Error preparing data for projected volume kube-api-access-fcpbr for pod openshift-network-diagnostics/network-check-target-sgwg2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:54.882526 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:54.882510 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr podName:34296475-0929-4b62-84ce-1666247486b3 nodeName:}" failed. No retries permitted until 2026-09-10 04:42:26.882497292 +0000 UTC m=+66.386330332 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcpbr" (UniqueName: "kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr") pod "network-check-target-sgwg2" (UID: "34296475-0929-4b62-84ce-1666247486b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 04:41:55.030763 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:55.030734 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:55.030939 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:55.030841 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:55.236771 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:55.236730 2572 generic.go:356] "Generic (PLEG): container finished" podID="195cb462-b600-46bc-ab74-21a87101050a" containerID="2524d3842fcc74643a1f470716b7018532a22241c6b17f836765042a137e93ac" exitCode=0 Sep 10 04:41:55.236771 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:55.236769 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg5jd" event={"ID":"195cb462-b600-46bc-ab74-21a87101050a","Type":"ContainerDied","Data":"2524d3842fcc74643a1f470716b7018532a22241c6b17f836765042a137e93ac"} Sep 10 04:41:56.030897 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:56.030856 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:56.031100 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:56.030856 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:56.031100 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:56.030967 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:56.031100 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:56.031070 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:56.242358 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:56.242324 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg5jd" event={"ID":"195cb462-b600-46bc-ab74-21a87101050a","Type":"ContainerStarted","Data":"7aef13daba33ba975b532c86e563949d91ccea68c8be284810dab78e750ede5a"} Sep 10 04:41:56.263681 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:56.263628 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sg5jd" podStartSLOduration=7.302054143 podStartE2EDuration="35.263611574s" totalImagesPullingTime="27.961557431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:41:56.261981382 +0000 UTC m=+35.765814442" watchObservedRunningTime="2026-09-10 04:41:56.263611574 +0000 UTC m=+35.767444662" Sep 10 04:41:57.030463 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:57.030422 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:57.030653 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:57.030550 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:41:58.030570 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:58.030529 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:41:58.031077 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:58.030534 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:41:58.031077 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:58.030664 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:41:58.031077 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:58.030713 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:41:59.030274 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:41:59.030232 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:41:59.030496 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:41:59.030357 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:42:00.030417 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:00.030376 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:42:00.030817 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:00.030391 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:42:00.030817 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:00.030524 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:42:00.030817 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:00.030623 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:42:01.031604 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:01.031568 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:42:01.031997 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:01.031701 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:42:02.030461 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:02.030407 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:42:02.030642 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:02.030472 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:42:02.030642 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:02.030535 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:42:02.030723 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:02.030643 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:42:03.031257 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:03.031217 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:42:03.031766 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:03.031374 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:42:04.030353 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:04.030312 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:42:04.030548 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:04.030321 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:42:04.030548 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:04.030416 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:42:04.030548 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:04.030518 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:42:05.030363 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:05.030319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:42:05.030780 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:05.030499 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:42:06.030583 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:06.030543 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:42:06.031056 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:06.030543 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:42:06.031056 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:06.030676 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sgwg2" podUID="34296475-0929-4b62-84ce-1666247486b3" Sep 10 04:42:06.031056 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:06.030729 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mh4x5" podUID="46bf0532-232f-423a-add2-7c0ff0a5a9a9" Sep 10 04:42:07.030467 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.030426 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:42:07.030642 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:07.030587 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n2dx5" podUID="169267a0-c3db-4d30-b1ab-1e70511f19c6" Sep 10 04:42:07.838002 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.837977 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeReady" Sep 10 04:42:07.838197 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.838142 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 04:42:07.896030 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.895993 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pcllp"] Sep 10 04:42:07.911201 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.911166 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8ffk8"] Sep 10 04:42:07.911376 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.911285 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:07.914085 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.914056 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nj7q4\"" Sep 10 04:42:07.914247 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.914190 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 04:42:07.914589 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.914572 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 04:42:07.923320 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.923296 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcllp"] Sep 10 04:42:07.923441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.923332 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8ffk8"] Sep 10 04:42:07.923441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.923429 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8ffk8" Sep 10 04:42:07.925940 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.925921 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 04:42:07.926122 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.926008 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d4254\"" Sep 10 04:42:07.926122 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.926020 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 04:42:07.926122 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.926061 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 04:42:07.993613 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:07.993578 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sncsj"] Sep 10 04:42:08.013555 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.013516 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sncsj"] Sep 10 04:42:08.013736 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.013669 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.016533 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.016511 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 04:42:08.016721 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.016696 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kf5vp\"" Sep 10 04:42:08.016822 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.016698 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 04:42:08.016822 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.016737 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 04:42:08.016822 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.016769 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 04:42:08.030227 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.030189 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:42:08.030227 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.030216 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:42:08.033223 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.033197 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 04:42:08.033641 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.033225 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gx84n\"" Sep 10 04:42:08.033641 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.033230 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 04:42:08.033641 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.033275 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 04:42:08.067872 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.067833 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqp4b\" (UniqueName: \"kubernetes.io/projected/c6812836-3390-4aa9-b1b3-b997d53e56fb-kube-api-access-kqp4b\") pod \"ingress-canary-8ffk8\" (UID: \"c6812836-3390-4aa9-b1b3-b997d53e56fb\") " pod="openshift-ingress-canary/ingress-canary-8ffk8" Sep 10 04:42:08.067872 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.067871 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da0728c-18d9-4d7b-9122-0fccd5a6c298-tmp-dir\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.068085 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.067904 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7da0728c-18d9-4d7b-9122-0fccd5a6c298-metrics-tls\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.068085 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.067973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssszj\" (UniqueName: \"kubernetes.io/projected/7da0728c-18d9-4d7b-9122-0fccd5a6c298-kube-api-access-ssszj\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.068085 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.068032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7da0728c-18d9-4d7b-9122-0fccd5a6c298-config-volume\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.068085 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.068074 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6812836-3390-4aa9-b1b3-b997d53e56fb-cert\") pod \"ingress-canary-8ffk8\" (UID: \"c6812836-3390-4aa9-b1b3-b997d53e56fb\") " pod="openshift-ingress-canary/ingress-canary-8ffk8" Sep 10 04:42:08.169330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169237 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7da0728c-18d9-4d7b-9122-0fccd5a6c298-metrics-tls\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.169330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssszj\" (UniqueName: \"kubernetes.io/projected/7da0728c-18d9-4d7b-9122-0fccd5a6c298-kube-api-access-ssszj\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.169330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169311 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc6b935a-1508-46df-9efe-ce6f1bd0da04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.169330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7da0728c-18d9-4d7b-9122-0fccd5a6c298-config-volume\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.169624 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169488 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc6b935a-1508-46df-9efe-ce6f1bd0da04-crio-socket\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.169624 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6812836-3390-4aa9-b1b3-b997d53e56fb-cert\") pod \"ingress-canary-8ffk8\" (UID: \"c6812836-3390-4aa9-b1b3-b997d53e56fb\") " pod="openshift-ingress-canary/ingress-canary-8ffk8" Sep 10 04:42:08.169624 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169583 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc6b935a-1508-46df-9efe-ce6f1bd0da04-data-volume\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.169624 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169613 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc6b935a-1508-46df-9efe-ce6f1bd0da04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.169831 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqp4b\" (UniqueName: \"kubernetes.io/projected/c6812836-3390-4aa9-b1b3-b997d53e56fb-kube-api-access-kqp4b\") pod \"ingress-canary-8ffk8\" (UID: \"c6812836-3390-4aa9-b1b3-b997d53e56fb\") " pod="openshift-ingress-canary/ingress-canary-8ffk8" Sep 10 04:42:08.169831 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169788 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da0728c-18d9-4d7b-9122-0fccd5a6c298-tmp-dir\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.169831 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169827 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzw4j\" (UniqueName: \"kubernetes.io/projected/cc6b935a-1508-46df-9efe-ce6f1bd0da04-kube-api-access-vzw4j\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.169989 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.169850 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7da0728c-18d9-4d7b-9122-0fccd5a6c298-config-volume\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.170078 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.170062 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7da0728c-18d9-4d7b-9122-0fccd5a6c298-tmp-dir\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.173625 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.173599 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7da0728c-18d9-4d7b-9122-0fccd5a6c298-metrics-tls\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.173752 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.173649 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6812836-3390-4aa9-b1b3-b997d53e56fb-cert\") pod \"ingress-canary-8ffk8\" (UID: \"c6812836-3390-4aa9-b1b3-b997d53e56fb\") " pod="openshift-ingress-canary/ingress-canary-8ffk8" Sep 10 04:42:08.178625 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.178597 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssszj\" (UniqueName: \"kubernetes.io/projected/7da0728c-18d9-4d7b-9122-0fccd5a6c298-kube-api-access-ssszj\") pod \"dns-default-pcllp\" (UID: \"7da0728c-18d9-4d7b-9122-0fccd5a6c298\") " pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.178754 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.178683 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqp4b\" (UniqueName: \"kubernetes.io/projected/c6812836-3390-4aa9-b1b3-b997d53e56fb-kube-api-access-kqp4b\") pod \"ingress-canary-8ffk8\" (UID: \"c6812836-3390-4aa9-b1b3-b997d53e56fb\") " pod="openshift-ingress-canary/ingress-canary-8ffk8" Sep 10 04:42:08.221208 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.221171 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:08.232165 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.232136 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8ffk8" Sep 10 04:42:08.270501 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.270459 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc6b935a-1508-46df-9efe-ce6f1bd0da04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.270608 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.270503 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc6b935a-1508-46df-9efe-ce6f1bd0da04-crio-socket\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.270608 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.270546 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc6b935a-1508-46df-9efe-ce6f1bd0da04-data-volume\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.270608 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.270574 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc6b935a-1508-46df-9efe-ce6f1bd0da04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.270773 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.270616 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzw4j\" (UniqueName: \"kubernetes.io/projected/cc6b935a-1508-46df-9efe-ce6f1bd0da04-kube-api-access-vzw4j\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.270829 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.270788 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc6b935a-1508-46df-9efe-ce6f1bd0da04-crio-socket\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.271145 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.271111 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc6b935a-1508-46df-9efe-ce6f1bd0da04-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.271319 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.271298 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc6b935a-1508-46df-9efe-ce6f1bd0da04-data-volume\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.272970 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.272948 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc6b935a-1508-46df-9efe-ce6f1bd0da04-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.279104 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.279081 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzw4j\" (UniqueName: \"kubernetes.io/projected/cc6b935a-1508-46df-9efe-ce6f1bd0da04-kube-api-access-vzw4j\") pod \"insights-runtime-extractor-sncsj\" (UID: \"cc6b935a-1508-46df-9efe-ce6f1bd0da04\") " pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.322087 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.322045 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sncsj" Sep 10 04:42:08.557920 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.557891 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pcllp"] Sep 10 04:42:08.561069 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:08.561041 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da0728c_18d9_4d7b_9122_0fccd5a6c298.slice/crio-aab23ec1439d458740a280c7a510153173feacd2762913d2b51ec2bf03a2ac28 WatchSource:0}: Error finding container aab23ec1439d458740a280c7a510153173feacd2762913d2b51ec2bf03a2ac28: Status 404 returned error can't find the container with id aab23ec1439d458740a280c7a510153173feacd2762913d2b51ec2bf03a2ac28 Sep 10 04:42:08.581567 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.581536 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8ffk8"] Sep 10 04:42:08.584397 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:08.584376 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6812836_3390_4aa9_b1b3_b997d53e56fb.slice/crio-bb5ce1803058efb8ae0f02ebb4bb99424663b50161b695c6c8f5ca63fcbbe9ed WatchSource:0}: Error finding container bb5ce1803058efb8ae0f02ebb4bb99424663b50161b695c6c8f5ca63fcbbe9ed: Status 404 returned error can't find the container with id bb5ce1803058efb8ae0f02ebb4bb99424663b50161b695c6c8f5ca63fcbbe9ed Sep 10 04:42:08.622257 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:08.622223 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sncsj"] Sep 10 04:42:08.625288 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:08.625257 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6b935a_1508_46df_9efe_ce6f1bd0da04.slice/crio-c46d38842cc3de764ec7fca924bffb737c598a8f846fde50b731df6f4958bd84 WatchSource:0}: Error finding container c46d38842cc3de764ec7fca924bffb737c598a8f846fde50b731df6f4958bd84: Status 404 returned error can't find the container with id c46d38842cc3de764ec7fca924bffb737c598a8f846fde50b731df6f4958bd84 Sep 10 04:42:09.030691 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:09.030654 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:42:09.034507 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:09.034476 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jksxr\"" Sep 10 04:42:09.034507 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:09.034502 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 04:42:09.274538 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:09.274499 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcllp" event={"ID":"7da0728c-18d9-4d7b-9122-0fccd5a6c298","Type":"ContainerStarted","Data":"aab23ec1439d458740a280c7a510153173feacd2762913d2b51ec2bf03a2ac28"} Sep 10 04:42:09.277860 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:09.277826 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sncsj" event={"ID":"cc6b935a-1508-46df-9efe-ce6f1bd0da04","Type":"ContainerStarted","Data":"a589df33f56d55d6953ca7beca022dad8217f58a5325859a93a50d357871d089"} Sep 10 04:42:09.278037 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:09.277894 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sncsj" event={"ID":"cc6b935a-1508-46df-9efe-ce6f1bd0da04","Type":"ContainerStarted","Data":"c46d38842cc3de764ec7fca924bffb737c598a8f846fde50b731df6f4958bd84"} Sep 10 04:42:09.280162 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:09.280129 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8ffk8" event={"ID":"c6812836-3390-4aa9-b1b3-b997d53e56fb","Type":"ContainerStarted","Data":"bb5ce1803058efb8ae0f02ebb4bb99424663b50161b695c6c8f5ca63fcbbe9ed"} Sep 10 04:42:10.211599 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.211562 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd"] Sep 10 04:42:10.226046 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.226007 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd" Sep 10 04:42:10.228721 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.228694 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vhrd6\"" Sep 10 04:42:10.228884 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.228739 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 04:42:10.231353 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.231324 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd"] Sep 10 04:42:10.310403 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.310368 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-44wrh"] Sep 10 04:42:10.324240 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.324201 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-44wrh"] Sep 10 04:42:10.324420 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.324352 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-44wrh" Sep 10 04:42:10.326979 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.326953 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 04:42:10.327133 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.327009 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-756kd\"" Sep 10 04:42:10.327133 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.327035 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 04:42:10.388626 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.388586 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/97a4470f-9ff5-446d-81b7-79288f79b652-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kvpdd\" (UID: \"97a4470f-9ff5-446d-81b7-79288f79b652\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd" Sep 10 04:42:10.490139 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.490104 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtpx\" (UniqueName: \"kubernetes.io/projected/01e9fb97-7654-4ee0-88f3-baeef295cfbe-kube-api-access-smtpx\") pod \"downloads-85d98c998f-44wrh\" (UID: \"01e9fb97-7654-4ee0-88f3-baeef295cfbe\") " pod="openshift-console/downloads-85d98c998f-44wrh" Sep 10 04:42:10.490351 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.490147 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/97a4470f-9ff5-446d-81b7-79288f79b652-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kvpdd\" (UID: \"97a4470f-9ff5-446d-81b7-79288f79b652\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd" Sep 10 04:42:10.490351 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:10.490244 2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 10 04:42:10.490351 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:10.490299 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97a4470f-9ff5-446d-81b7-79288f79b652-tls-certificates podName:97a4470f-9ff5-446d-81b7-79288f79b652 nodeName:}" failed. No retries permitted until 2026-09-10 04:42:10.990284395 +0000 UTC m=+50.494117433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/97a4470f-9ff5-446d-81b7-79288f79b652-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-kvpdd" (UID: "97a4470f-9ff5-446d-81b7-79288f79b652") : secret "prometheus-operator-admission-webhook-tls" not found Sep 10 04:42:10.591119 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.591081 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smtpx\" (UniqueName: \"kubernetes.io/projected/01e9fb97-7654-4ee0-88f3-baeef295cfbe-kube-api-access-smtpx\") pod \"downloads-85d98c998f-44wrh\" (UID: \"01e9fb97-7654-4ee0-88f3-baeef295cfbe\") " pod="openshift-console/downloads-85d98c998f-44wrh" Sep 10 04:42:10.600616 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.600579 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smtpx\" (UniqueName: \"kubernetes.io/projected/01e9fb97-7654-4ee0-88f3-baeef295cfbe-kube-api-access-smtpx\") pod \"downloads-85d98c998f-44wrh\" (UID: \"01e9fb97-7654-4ee0-88f3-baeef295cfbe\") " pod="openshift-console/downloads-85d98c998f-44wrh" Sep 10 04:42:10.635271 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.635221 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-44wrh" Sep 10 04:42:10.993584 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.993547 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/97a4470f-9ff5-446d-81b7-79288f79b652-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kvpdd\" (UID: \"97a4470f-9ff5-446d-81b7-79288f79b652\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd" Sep 10 04:42:10.996826 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:10.996777 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/97a4470f-9ff5-446d-81b7-79288f79b652-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-kvpdd\" (UID: \"97a4470f-9ff5-446d-81b7-79288f79b652\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd" Sep 10 04:42:11.138512 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:11.138471 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd" Sep 10 04:42:11.497946 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:11.497850 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd"] Sep 10 04:42:11.501985 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:11.501917 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a4470f_9ff5_446d_81b7_79288f79b652.slice/crio-a4cabd149223074f84a649262251614fa3cb609f5ffd19c27ab45315ec02e65c WatchSource:0}: Error finding container a4cabd149223074f84a649262251614fa3cb609f5ffd19c27ab45315ec02e65c: Status 404 returned error can't find the container with id a4cabd149223074f84a649262251614fa3cb609f5ffd19c27ab45315ec02e65c Sep 10 04:42:11.503601 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:11.503569 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-44wrh"] Sep 10 04:42:11.507697 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:11.507666 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e9fb97_7654_4ee0_88f3_baeef295cfbe.slice/crio-322439b6e9985fa80c1e5e290f3d7fbb1e49b77830635d7b88922e027e73ce4c WatchSource:0}: Error finding container 322439b6e9985fa80c1e5e290f3d7fbb1e49b77830635d7b88922e027e73ce4c: Status 404 returned error can't find the container with id 322439b6e9985fa80c1e5e290f3d7fbb1e49b77830635d7b88922e027e73ce4c Sep 10 04:42:12.295110 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:12.295007 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sncsj" event={"ID":"cc6b935a-1508-46df-9efe-ce6f1bd0da04","Type":"ContainerStarted","Data":"9987c7dd30b71c83be45d3057da6290e82ea313985273fdec439b9424fc4778a"} Sep 10 04:42:12.297231 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:12.297195 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8ffk8" event={"ID":"c6812836-3390-4aa9-b1b3-b997d53e56fb","Type":"ContainerStarted","Data":"29aaa0ba08e1b9b7b4e06e76634281896885610ecc7b130ed0996d361a3e9437"} Sep 10 04:42:12.302897 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:12.302859 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcllp" event={"ID":"7da0728c-18d9-4d7b-9122-0fccd5a6c298","Type":"ContainerStarted","Data":"3511cd7f9d8f54b795c21c2962b3d7e004996f4cac28d60507bf1593c69ab274"} Sep 10 04:42:12.303034 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:12.302926 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pcllp" event={"ID":"7da0728c-18d9-4d7b-9122-0fccd5a6c298","Type":"ContainerStarted","Data":"22b8fc904ac085e1aa367ed89a8eeb2695405f63b1c67d4b636243c5ad47a8ae"} Sep 10 04:42:12.303110 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:12.303032 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:12.304667 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:12.304643 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-44wrh" event={"ID":"01e9fb97-7654-4ee0-88f3-baeef295cfbe","Type":"ContainerStarted","Data":"322439b6e9985fa80c1e5e290f3d7fbb1e49b77830635d7b88922e027e73ce4c"} Sep 10 04:42:12.316484 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:12.313947 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd" event={"ID":"97a4470f-9ff5-446d-81b7-79288f79b652","Type":"ContainerStarted","Data":"a4cabd149223074f84a649262251614fa3cb609f5ffd19c27ab45315ec02e65c"} Sep 10 04:42:12.316484 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:12.314802 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8ffk8" podStartSLOduration=2.672899754 podStartE2EDuration="5.314767083s" totalImagesPullingTime="2.641867329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:12.314344123 +0000 UTC m=+51.818177184" watchObservedRunningTime="2026-09-10 04:42:12.314767083 +0000 UTC m=+51.818600146" Sep 10 04:42:12.337622 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:12.337559 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pcllp" podStartSLOduration=2.678854726 podStartE2EDuration="5.337535403s" totalImagesPullingTime="2.658680677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:12.337012484 +0000 UTC m=+51.840845545" watchObservedRunningTime="2026-09-10 04:42:12.337535403 +0000 UTC m=+51.841368465" Sep 10 04:42:13.320928 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:13.320841 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sncsj" event={"ID":"cc6b935a-1508-46df-9efe-ce6f1bd0da04","Type":"ContainerStarted","Data":"b26d7c53ed33f4786088657848c64fce2269c662b6ab61c9061a5ef0f414ec28"} Sep 10 04:42:13.322775 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:13.322735 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd" event={"ID":"97a4470f-9ff5-446d-81b7-79288f79b652","Type":"ContainerStarted","Data":"5eff1e66d2a1943efd5e6edf634cccb266ce8e42266429a218174a8eabe530a4"} Sep 10 04:42:13.322950 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:13.322928 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd" Sep 10 04:42:13.328524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:13.328496 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd" Sep 10 04:42:13.339114 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:13.339060 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sncsj" podStartSLOduration=2.2433690139999998 podStartE2EDuration="6.339040226s" totalImagesPullingTime="4.095671212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:42:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:13.337826575 +0000 UTC m=+52.841659650" watchObservedRunningTime="2026-09-10 04:42:13.339040226 +0000 UTC m=+52.842873289" Sep 10 04:42:13.354781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:13.354686 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-kvpdd" podStartSLOduration=1.845430485 podStartE2EDuration="3.354672851s" totalImagesPullingTime="1.509242366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:13.353738309 +0000 UTC m=+52.857571373" watchObservedRunningTime="2026-09-10 04:42:13.354672851 +0000 UTC m=+52.858505908" Sep 10 04:42:14.268997 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.268956 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-k4vcd"] Sep 10 04:42:14.272505 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.272471 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.276790 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.276758 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 04:42:14.276790 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.276773 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 04:42:14.277015 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.276827 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 04:42:14.277015 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.276836 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 04:42:14.277015 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.276772 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 04:42:14.277174 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.277048 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9hz5t\"" Sep 10 04:42:14.281506 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.281475 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-k4vcd"] Sep 10 04:42:14.421794 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.421751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2137982-cc72-482c-bcdd-1520a94294e4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.422265 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.421801 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2137982-cc72-482c-bcdd-1520a94294e4-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.422265 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.421937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2137982-cc72-482c-bcdd-1520a94294e4-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.422265 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.422032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j62lt\" (UniqueName: \"kubernetes.io/projected/d2137982-cc72-482c-bcdd-1520a94294e4-kube-api-access-j62lt\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.523370 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.523284 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j62lt\" (UniqueName: \"kubernetes.io/projected/d2137982-cc72-482c-bcdd-1520a94294e4-kube-api-access-j62lt\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.523654 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.523392 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2137982-cc72-482c-bcdd-1520a94294e4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.523654 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.523435 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2137982-cc72-482c-bcdd-1520a94294e4-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.523654 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.523489 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2137982-cc72-482c-bcdd-1520a94294e4-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.524268 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.524238 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2137982-cc72-482c-bcdd-1520a94294e4-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.526303 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.526273 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2137982-cc72-482c-bcdd-1520a94294e4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.526413 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.526324 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2137982-cc72-482c-bcdd-1520a94294e4-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.532953 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.532917 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j62lt\" (UniqueName: \"kubernetes.io/projected/d2137982-cc72-482c-bcdd-1520a94294e4-kube-api-access-j62lt\") pod \"prometheus-operator-6f4946db56-k4vcd\" (UID: \"d2137982-cc72-482c-bcdd-1520a94294e4\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.584697 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.584666 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" Sep 10 04:42:14.867872 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:14.867789 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-k4vcd"] Sep 10 04:42:14.870986 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:14.870942 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2137982_cc72_482c_bcdd_1520a94294e4.slice/crio-f404146e8eee59f12891cb741d4291bb09cf0c8116d7fea9ed749891b1ae9955 WatchSource:0}: Error finding container f404146e8eee59f12891cb741d4291bb09cf0c8116d7fea9ed749891b1ae9955: Status 404 returned error can't find the container with id f404146e8eee59f12891cb741d4291bb09cf0c8116d7fea9ed749891b1ae9955 Sep 10 04:42:15.332111 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:15.332068 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" event={"ID":"d2137982-cc72-482c-bcdd-1520a94294e4","Type":"ContainerStarted","Data":"f404146e8eee59f12891cb741d4291bb09cf0c8116d7fea9ed749891b1ae9955"} Sep 10 04:42:17.342520 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:17.342476 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" event={"ID":"d2137982-cc72-482c-bcdd-1520a94294e4","Type":"ContainerStarted","Data":"ac962ff01b005d22f2d29a88fba12ed25e7df89c6427e144f41d2b0eea6f3006"} Sep 10 04:42:17.342989 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:17.342549 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" event={"ID":"d2137982-cc72-482c-bcdd-1520a94294e4","Type":"ContainerStarted","Data":"7fb3f617d3aef8a96e1c390990a1d649284f60a68a2ecd2bb616980f8465f2a7"} Sep 10 04:42:17.359722 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:17.359665 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-k4vcd" podStartSLOduration=1.914604068 podStartE2EDuration="3.359643504s" totalImagesPullingTime="1.445039436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:17.358550267 +0000 UTC m=+56.862383329" watchObservedRunningTime="2026-09-10 04:42:17.359643504 +0000 UTC m=+56.863476568" Sep 10 04:42:19.671741 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.671707 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x"] Sep 10 04:42:19.675437 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.675411 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:19.678538 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.678513 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 10 04:42:19.678694 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.678538 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 10 04:42:19.678970 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.678949 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dhnvv\"" Sep 10 04:42:19.685791 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.685762 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x"] Sep 10 04:42:19.727170 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.727138 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5"] Sep 10 04:42:19.730542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.730507 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.734181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.734085 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 10 04:42:19.734181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.734163 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 10 04:42:19.734516 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.734414 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 10 04:42:19.734680 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.734660 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-25t8d\"" Sep 10 04:42:19.747000 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.746974 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2lbl5"] Sep 10 04:42:19.749393 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.749369 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.754320 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.754283 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 04:42:19.754503 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.754482 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9hw97\"" Sep 10 04:42:19.754660 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.754640 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 04:42:19.754749 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.754596 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 04:42:19.755422 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.755355 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5"] Sep 10 04:42:19.770555 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.770518 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.770555 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.770566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xt8w\" (UniqueName: \"kubernetes.io/projected/d60fe704-87f9-4b56-8196-4354e0199fb3-kube-api-access-7xt8w\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:19.770851 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.770598 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4x7z\" (UniqueName: \"kubernetes.io/projected/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-api-access-h4x7z\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.770851 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.770622 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-wtmp\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.770851 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.770649 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.770851 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.770679 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-textfile\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.770851 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.770707 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.770851 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.770778 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d60fe704-87f9-4b56-8196-4354e0199fb3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:19.770851 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.770835 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400d2682-da56-4ff8-adc3-0d514bb7f1be-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.771192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.770873 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07f9db3c-da60-4f88-8327-61bd6936ecc3-sys\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.771192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.770946 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07f9db3c-da60-4f88-8327-61bd6936ecc3-root\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.771192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.771003 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krnlb\" (UniqueName: \"kubernetes.io/projected/07f9db3c-da60-4f88-8327-61bd6936ecc3-kube-api-access-krnlb\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.771192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.771029 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d60fe704-87f9-4b56-8196-4354e0199fb3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:19.771192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.771056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f9db3c-da60-4f88-8327-61bd6936ecc3-metrics-client-ca\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.771192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.771080 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.771192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.771106 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-tls\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.771192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.771163 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/400d2682-da56-4ff8-adc3-0d514bb7f1be-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.771192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.771196 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.771549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.771221 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d60fe704-87f9-4b56-8196-4354e0199fb3-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:19.871896 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.871849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.872098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.871907 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xt8w\" (UniqueName: \"kubernetes.io/projected/d60fe704-87f9-4b56-8196-4354e0199fb3-kube-api-access-7xt8w\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:19.872098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.871936 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4x7z\" (UniqueName: \"kubernetes.io/projected/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-api-access-h4x7z\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.872098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.871961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-wtmp\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.872098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.871987 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.872098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872017 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-textfile\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.872098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872049 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.872098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872078 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d60fe704-87f9-4b56-8196-4354e0199fb3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:19.872441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872104 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400d2682-da56-4ff8-adc3-0d514bb7f1be-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.872441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872129 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07f9db3c-da60-4f88-8327-61bd6936ecc3-sys\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.872441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872161 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07f9db3c-da60-4f88-8327-61bd6936ecc3-root\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.872441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872159 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-wtmp\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.872441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krnlb\" (UniqueName: \"kubernetes.io/projected/07f9db3c-da60-4f88-8327-61bd6936ecc3-kube-api-access-krnlb\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.872441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872242 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d60fe704-87f9-4b56-8196-4354e0199fb3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:19.872441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872268 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f9db3c-da60-4f88-8327-61bd6936ecc3-metrics-client-ca\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.872441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872290 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.872441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872315 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-tls\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.872441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/400d2682-da56-4ff8-adc3-0d514bb7f1be-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.872441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872377 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.873015 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d60fe704-87f9-4b56-8196-4354e0199fb3-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:19.873015 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872789 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.873015 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872789 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.873015 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872857 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/400d2682-da56-4ff8-adc3-0d514bb7f1be-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.873015 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.872919 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-textfile\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.873265 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.873223 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/07f9db3c-da60-4f88-8327-61bd6936ecc3-root\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.873320 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.873275 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07f9db3c-da60-4f88-8327-61bd6936ecc3-sys\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.873320 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.873299 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07f9db3c-da60-4f88-8327-61bd6936ecc3-metrics-client-ca\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.873422 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:19.873394 2572 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 10 04:42:19.873422 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.873397 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d60fe704-87f9-4b56-8196-4354e0199fb3-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:19.873543 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:19.873473 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d60fe704-87f9-4b56-8196-4354e0199fb3-openshift-state-metrics-tls podName:d60fe704-87f9-4b56-8196-4354e0199fb3 nodeName:}" failed. No retries permitted until 2026-09-10 04:42:20.373435524 +0000 UTC m=+59.877268580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d60fe704-87f9-4b56-8196-4354e0199fb3-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-pgx4x" (UID: "d60fe704-87f9-4b56-8196-4354e0199fb3") : secret "openshift-state-metrics-tls" not found Sep 10 04:42:19.873756 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.873710 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/400d2682-da56-4ff8-adc3-0d514bb7f1be-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.875746 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.875697 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.875871 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.875808 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/07f9db3c-da60-4f88-8327-61bd6936ecc3-node-exporter-tls\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.875871 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.875848 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.876281 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.876252 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d60fe704-87f9-4b56-8196-4354e0199fb3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:19.876387 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.876293 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.889302 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.889264 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4x7z\" (UniqueName: \"kubernetes.io/projected/400d2682-da56-4ff8-adc3-0d514bb7f1be-kube-api-access-h4x7z\") pod \"kube-state-metrics-74cb57df7-6t7c5\" (UID: \"400d2682-da56-4ff8-adc3-0d514bb7f1be\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:19.889826 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.889804 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krnlb\" (UniqueName: \"kubernetes.io/projected/07f9db3c-da60-4f88-8327-61bd6936ecc3-kube-api-access-krnlb\") pod \"node-exporter-2lbl5\" (UID: \"07f9db3c-da60-4f88-8327-61bd6936ecc3\") " pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:19.889952 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:19.889828 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xt8w\" (UniqueName: \"kubernetes.io/projected/d60fe704-87f9-4b56-8196-4354e0199fb3-kube-api-access-7xt8w\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:20.042076 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.042034 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" Sep 10 04:42:20.060649 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.060609 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2lbl5" Sep 10 04:42:20.073151 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:20.073114 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f9db3c_da60_4f88_8327_61bd6936ecc3.slice/crio-baa4b5a3085a79d2a99af598b688c5d6bfc5ebd52412473a6e99ecbc573c340a WatchSource:0}: Error finding container baa4b5a3085a79d2a99af598b688c5d6bfc5ebd52412473a6e99ecbc573c340a: Status 404 returned error can't find the container with id baa4b5a3085a79d2a99af598b688c5d6bfc5ebd52412473a6e99ecbc573c340a Sep 10 04:42:20.228848 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.228819 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q7lb4" Sep 10 04:42:20.327972 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.327878 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5"] Sep 10 04:42:20.332127 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:20.332093 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400d2682_da56_4ff8_adc3_0d514bb7f1be.slice/crio-b28b9c85e0bcaf2b81c0d2869f7ed3ac66f69ccdaef8112caa1b717bf4bc2dd4 WatchSource:0}: Error finding container b28b9c85e0bcaf2b81c0d2869f7ed3ac66f69ccdaef8112caa1b717bf4bc2dd4: Status 404 returned error can't find the container with id b28b9c85e0bcaf2b81c0d2869f7ed3ac66f69ccdaef8112caa1b717bf4bc2dd4 Sep 10 04:42:20.353838 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.353779 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2lbl5" event={"ID":"07f9db3c-da60-4f88-8327-61bd6936ecc3","Type":"ContainerStarted","Data":"baa4b5a3085a79d2a99af598b688c5d6bfc5ebd52412473a6e99ecbc573c340a"} Sep 10 04:42:20.355173 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.355146 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" event={"ID":"400d2682-da56-4ff8-adc3-0d514bb7f1be","Type":"ContainerStarted","Data":"b28b9c85e0bcaf2b81c0d2869f7ed3ac66f69ccdaef8112caa1b717bf4bc2dd4"} Sep 10 04:42:20.377006 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.376967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d60fe704-87f9-4b56-8196-4354e0199fb3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:20.379496 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.379438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d60fe704-87f9-4b56-8196-4354e0199fb3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pgx4x\" (UID: \"d60fe704-87f9-4b56-8196-4354e0199fb3\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:20.587571 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.587484 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" Sep 10 04:42:20.734744 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.734710 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 04:42:20.739500 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.739469 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.743655 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.743171 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 04:42:20.743655 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.743412 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 04:42:20.744287 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.743946 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 04:42:20.744287 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.744008 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 04:42:20.744287 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.744015 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 04:42:20.744287 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.744041 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 04:42:20.744287 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.744199 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 04:42:20.744562 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.744400 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jxftf\"" Sep 10 04:42:20.744562 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.744466 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 04:42:20.744562 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.744405 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 04:42:20.760293 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.760259 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 04:42:20.780899 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.780857 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.780899 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.780912 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.781182 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.780969 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.781182 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.780995 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.781182 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.781029 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbgd\" (UniqueName: \"kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-kube-api-access-5dbgd\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.781182 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.781065 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.781182 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.781136 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.781567 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.781272 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.781567 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.781317 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-web-config\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.781567 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.781346 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.781567 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.781370 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-out\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.781567 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.781416 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.781567 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.781545 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.873362 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.873234 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x"] Sep 10 04:42:20.876143 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:20.876074 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60fe704_87f9_4b56_8196_4354e0199fb3.slice/crio-2b32a0b650992f6b4ef0ce51e37f799e68ab327f1efaecbff927b6ce697c4b7e WatchSource:0}: Error finding container 2b32a0b650992f6b4ef0ce51e37f799e68ab327f1efaecbff927b6ce697c4b7e: Status 404 returned error can't find the container with id 2b32a0b650992f6b4ef0ce51e37f799e68ab327f1efaecbff927b6ce697c4b7e Sep 10 04:42:20.882534 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882507 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.882534 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882539 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.882728 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbgd\" (UniqueName: \"kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-kube-api-access-5dbgd\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.882728 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882585 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.882728 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.882728 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.882728 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882693 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-web-config\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.882728 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:42:20.882722 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-trusted-ca-bundle podName:0d0c441d-8eca-4011-8c8f-c0c8b25750cd nodeName:}" failed. No retries permitted until 2026-09-10 04:42:21.382699234 +0000 UTC m=+60.886532291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd") : configmap references non-existent config key: ca-bundle.crt Sep 10 04:42:20.883197 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882762 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.883197 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882794 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-out\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.883197 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882819 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.883197 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.883197 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882899 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.883197 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.882932 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.883564 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.883306 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.885936 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.885905 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-web-config\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.886896 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.886142 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.886896 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.886857 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.888364 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.888325 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.888676 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.888631 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-out\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.889026 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.888844 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.890895 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.890854 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.891433 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.891268 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.891608 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.891467 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.893945 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.893905 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:20.896150 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:20.895998 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbgd\" (UniqueName: \"kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-kube-api-access-5dbgd\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:21.164752 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.164661 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dcd4c9476-hzwws"] Sep 10 04:42:21.168128 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.168101 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.170819 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.170793 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 04:42:21.171216 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.171196 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-v4tq8\"" Sep 10 04:42:21.171322 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.171237 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 04:42:21.171378 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.171347 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 04:42:21.171378 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.171366 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 04:42:21.171716 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.171696 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 04:42:21.178909 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.178879 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcd4c9476-hzwws"] Sep 10 04:42:21.184623 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.184591 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjwjj\" (UniqueName: \"kubernetes.io/projected/a89551a4-3391-4bde-a5df-c404fb2c1a6c-kube-api-access-cjwjj\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.184761 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.184635 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-service-ca\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.184761 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.184686 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-config\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.184761 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.184706 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-serving-cert\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.184761 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.184734 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-oauth-config\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.184968 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.184789 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-oauth-serving-cert\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.285369 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.285327 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjwjj\" (UniqueName: \"kubernetes.io/projected/a89551a4-3391-4bde-a5df-c404fb2c1a6c-kube-api-access-cjwjj\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.285557 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.285391 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-service-ca\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.285557 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.285492 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-config\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.285695 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.285643 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-serving-cert\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.285744 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.285708 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-oauth-config\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.286169 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.285915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-oauth-serving-cert\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.286953 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.286916 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-service-ca\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.287309 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.287284 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-config\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.287387 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.287336 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-oauth-serving-cert\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.292746 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.292718 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-serving-cert\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.292950 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.292747 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-oauth-config\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.297122 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.297095 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjwjj\" (UniqueName: \"kubernetes.io/projected/a89551a4-3391-4bde-a5df-c404fb2c1a6c-kube-api-access-cjwjj\") pod \"console-dcd4c9476-hzwws\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.363060 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.363021 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" event={"ID":"d60fe704-87f9-4b56-8196-4354e0199fb3","Type":"ContainerStarted","Data":"0fe956f48c2b5f247b4d403f88fbf4e2e465c73de8b49b37cd5ac260fc14aaee"} Sep 10 04:42:21.363245 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.363129 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" event={"ID":"d60fe704-87f9-4b56-8196-4354e0199fb3","Type":"ContainerStarted","Data":"338fb30e5d1272060bb741a8c337c89155bc05256b77e39fd2d248915caeb8af"} Sep 10 04:42:21.363245 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.363159 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" event={"ID":"d60fe704-87f9-4b56-8196-4354e0199fb3","Type":"ContainerStarted","Data":"2b32a0b650992f6b4ef0ce51e37f799e68ab327f1efaecbff927b6ce697c4b7e"} Sep 10 04:42:21.386824 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.386777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:21.387790 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.387760 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:21.479011 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.478928 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:42:21.655282 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.655246 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jxftf\"" Sep 10 04:42:21.662637 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:21.662610 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:42:22.325245 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.325214 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pcllp" Sep 10 04:42:22.751937 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.751900 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55d58df5db-tzss8"] Sep 10 04:42:22.755865 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.755836 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.758784 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.758756 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fi2tqmklkgpsj\"" Sep 10 04:42:22.758999 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.758931 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-b9bc6\"" Sep 10 04:42:22.759199 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.759182 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 04:42:22.759199 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.759191 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 04:42:22.759428 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.759411 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 04:42:22.759589 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.759458 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 04:42:22.759589 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.759512 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 04:42:22.769594 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.769567 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55d58df5db-tzss8"] Sep 10 04:42:22.799467 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.799406 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.799660 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.799477 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.799660 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.799513 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.799660 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.799580 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-metrics-client-ca\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.799660 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.799632 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-tls\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.799660 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.799654 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4qpg\" (UniqueName: \"kubernetes.io/projected/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-kube-api-access-h4qpg\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.799907 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.799681 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.799907 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.799697 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-grpc-tls\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.900825 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.900783 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.900825 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.900829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.901084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.900862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.901084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.900923 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-metrics-client-ca\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.901084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.900978 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-tls\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.901084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.901005 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4qpg\" (UniqueName: \"kubernetes.io/projected/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-kube-api-access-h4qpg\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.901084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.901028 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.901084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.901051 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-grpc-tls\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.901950 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.901898 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-metrics-client-ca\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.903947 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.903896 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.904324 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.904296 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-grpc-tls\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.904424 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.904301 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.904424 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.904379 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.904548 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.904516 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.905066 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.905042 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-secret-thanos-querier-tls\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:22.910475 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:22.910426 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4qpg\" (UniqueName: \"kubernetes.io/projected/7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a-kube-api-access-h4qpg\") pod \"thanos-querier-55d58df5db-tzss8\" (UID: \"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a\") " pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:23.066671 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:23.066576 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:24.186798 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.186759 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-77b8b9cc78-d2hfw"] Sep 10 04:42:24.204405 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.204373 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77b8b9cc78-d2hfw"] Sep 10 04:42:24.204596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.204531 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.207268 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.207237 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 04:42:24.207425 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.207241 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 04:42:24.208612 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.208589 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zzfzt\"" Sep 10 04:42:24.208742 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.208674 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 04:42:24.208811 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.208588 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bifgalq23t6sr\"" Sep 10 04:42:24.220340 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.220285 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 04:42:24.314034 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.314000 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-metrics-server-audit-profiles\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.314205 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.314047 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.314205 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.314095 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-audit-log\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.314205 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.314132 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-secret-metrics-server-client-certs\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.314205 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.314188 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-client-ca-bundle\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.314395 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.314280 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-secret-metrics-server-tls\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.314395 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.314318 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mjf5\" (UniqueName: \"kubernetes.io/projected/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-kube-api-access-5mjf5\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.394335 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.394299 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws"] Sep 10 04:42:24.410301 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.410269 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" Sep 10 04:42:24.413284 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.413257 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v49sd\"" Sep 10 04:42:24.413523 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.413502 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 04:42:24.414919 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.414895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-secret-metrics-server-tls\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.415041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.414942 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mjf5\" (UniqueName: \"kubernetes.io/projected/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-kube-api-access-5mjf5\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.415041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.414993 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-metrics-server-audit-profiles\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.415041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.415031 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.415251 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.415062 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-audit-log\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.415251 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.415094 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-secret-metrics-server-client-certs\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.415251 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.415127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-client-ca-bundle\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.415942 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.415919 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.418753 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.417977 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-metrics-server-audit-profiles\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.418753 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.418173 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-audit-log\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.420306 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.419618 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-secret-metrics-server-tls\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.421180 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.421156 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws"] Sep 10 04:42:24.423529 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.422919 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-client-ca-bundle\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.424509 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.424475 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-secret-metrics-server-client-certs\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.432360 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.432333 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mjf5\" (UniqueName: \"kubernetes.io/projected/ea31ddbd-8d97-440e-a793-a56f5fc9ab12-kube-api-access-5mjf5\") pod \"metrics-server-77b8b9cc78-d2hfw\" (UID: \"ea31ddbd-8d97-440e-a793-a56f5fc9ab12\") " pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.515832 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.515769 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f193ed6-cb83-4049-8589-e5e8513606fe-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-q9fws\" (UID: \"2f193ed6-cb83-4049-8589-e5e8513606fe\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" Sep 10 04:42:24.516019 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.515867 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brmjn\" (UniqueName: \"kubernetes.io/projected/2f193ed6-cb83-4049-8589-e5e8513606fe-kube-api-access-brmjn\") pod \"monitoring-plugin-85bbbbbf4-q9fws\" (UID: \"2f193ed6-cb83-4049-8589-e5e8513606fe\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" Sep 10 04:42:24.516019 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.516012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:24.617412 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.617374 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f193ed6-cb83-4049-8589-e5e8513606fe-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-q9fws\" (UID: \"2f193ed6-cb83-4049-8589-e5e8513606fe\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" Sep 10 04:42:24.617653 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.617432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brmjn\" (UniqueName: \"kubernetes.io/projected/2f193ed6-cb83-4049-8589-e5e8513606fe-kube-api-access-brmjn\") pod \"monitoring-plugin-85bbbbbf4-q9fws\" (UID: \"2f193ed6-cb83-4049-8589-e5e8513606fe\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" Sep 10 04:42:24.620139 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.620116 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f193ed6-cb83-4049-8589-e5e8513606fe-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-q9fws\" (UID: \"2f193ed6-cb83-4049-8589-e5e8513606fe\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" Sep 10 04:42:24.625725 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.625697 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brmjn\" (UniqueName: \"kubernetes.io/projected/2f193ed6-cb83-4049-8589-e5e8513606fe-kube-api-access-brmjn\") pod \"monitoring-plugin-85bbbbbf4-q9fws\" (UID: \"2f193ed6-cb83-4049-8589-e5e8513606fe\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" Sep 10 04:42:24.742145 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:24.742111 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" Sep 10 04:42:25.890486 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.890425 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 04:42:25.894784 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.894751 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.897546 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.897516 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 04:42:25.897968 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.897935 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 04:42:25.897968 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.897957 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 04:42:25.898167 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.898005 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 04:42:25.898167 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.897956 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4f2wj\"" Sep 10 04:42:25.898289 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.898269 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6nh906m7lfq4a\"" Sep 10 04:42:25.898357 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.898305 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 04:42:25.900347 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.899314 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 04:42:25.900347 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.899567 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 04:42:25.900347 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.899785 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 04:42:25.900347 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.899973 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 04:42:25.900347 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.900144 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 04:42:25.901700 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.901157 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 04:42:25.901700 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.901343 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 04:42:25.901700 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.901575 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 04:42:25.903128 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.903097 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 04:42:25.908569 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.908463 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 04:42:25.930541 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930270 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-config\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.930541 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930324 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.930541 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930350 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.930541 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930387 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.930541 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930411 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.930541 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930501 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.930541 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930544 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930575 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930661 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930699 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930726 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930772 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-config-out\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930794 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930815 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930837 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930865 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-web-config\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:25.931007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:25.930915 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vf4h\" (UniqueName: \"kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-kube-api-access-4vf4h\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.031737 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.031791 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.031823 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.031844 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-config-out\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.031869 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.031891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.031913 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.031945 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.031975 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-web-config\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vf4h\" (UniqueName: \"kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-kube-api-access-4vf4h\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032081 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-config\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032110 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032134 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032176 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032202 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032226 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.032549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032268 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.033846 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032302 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.033846 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032347 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.033846 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032409 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.033846 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.032769 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.033846 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.033568 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.034220 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.034062 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.034406 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.034382 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.035439 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.035112 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.035787 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.035756 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.036651 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.036626 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.037398 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.037376 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.037685 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.037659 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.038172 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.038150 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-config\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.038739 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.038690 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.039019 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.038992 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-config-out\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.040131 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.040088 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.040131 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.040116 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.040284 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.040251 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.040284 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.040265 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.040419 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.040398 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.040526 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.040427 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.040526 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.040509 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-web-config\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.044220 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.044191 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vf4h\" (UniqueName: \"kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-kube-api-access-4vf4h\") pod \"prometheus-k8s-0\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.211398 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.211173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:26.741554 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.741499 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:42:26.744506 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.744473 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 04:42:26.754757 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.754724 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46bf0532-232f-423a-add2-7c0ff0a5a9a9-original-pull-secret\") pod \"global-pull-secret-syncer-mh4x5\" (UID: \"46bf0532-232f-423a-add2-7c0ff0a5a9a9\") " pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:42:26.842681 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.842644 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:42:26.845539 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.845508 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 04:42:26.856059 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.856029 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169267a0-c3db-4d30-b1ab-1e70511f19c6-metrics-certs\") pod \"network-metrics-daemon-n2dx5\" (UID: \"169267a0-c3db-4d30-b1ab-1e70511f19c6\") " pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:42:26.940012 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.939967 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mh4x5" Sep 10 04:42:26.943396 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.943118 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpbr\" (UniqueName: \"kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr\") pod \"network-check-target-sgwg2\" (UID: \"34296475-0929-4b62-84ce-1666247486b3\") " pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:42:26.946759 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.946726 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 04:42:26.956306 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.956277 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 04:42:26.967100 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:26.967063 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcpbr\" (UniqueName: \"kubernetes.io/projected/34296475-0929-4b62-84ce-1666247486b3-kube-api-access-fcpbr\") pod \"network-check-target-sgwg2\" (UID: \"34296475-0929-4b62-84ce-1666247486b3\") " pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:42:27.044664 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:27.044583 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jksxr\"" Sep 10 04:42:27.052868 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:27.052840 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n2dx5" Sep 10 04:42:27.247852 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:27.247816 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gx84n\"" Sep 10 04:42:27.256022 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:27.255986 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:42:29.287187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.286542 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws"] Sep 10 04:42:29.295181 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:29.294649 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f193ed6_cb83_4049_8589_e5e8513606fe.slice/crio-ed0b0e0c07c2ebb495ba4421b72dd13269f13ad33f08df65486e1ff2b3ddf992 WatchSource:0}: Error finding container ed0b0e0c07c2ebb495ba4421b72dd13269f13ad33f08df65486e1ff2b3ddf992: Status 404 returned error can't find the container with id ed0b0e0c07c2ebb495ba4421b72dd13269f13ad33f08df65486e1ff2b3ddf992 Sep 10 04:42:29.313767 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.313546 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sgwg2"] Sep 10 04:42:29.358327 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.358032 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcd4c9476-hzwws"] Sep 10 04:42:29.358600 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:29.358560 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34296475_0929_4b62_84ce_1666247486b3.slice/crio-531f7c12cd057b7fde1370d7d8c2e640db46e3a9d49055b396126d26d9dc9fa5 WatchSource:0}: Error finding container 531f7c12cd057b7fde1370d7d8c2e640db46e3a9d49055b396126d26d9dc9fa5: Status 404 returned error can't find the container with id 531f7c12cd057b7fde1370d7d8c2e640db46e3a9d49055b396126d26d9dc9fa5 Sep 10 04:42:29.367912 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.366300 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77b8b9cc78-d2hfw"] Sep 10 04:42:29.367912 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:29.366698 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89551a4_3391_4bde_a5df_c404fb2c1a6c.slice/crio-3d210f796b41316414aef07d873143143d3d4307a526be8842f733ee633f04b0 WatchSource:0}: Error finding container 3d210f796b41316414aef07d873143143d3d4307a526be8842f733ee633f04b0: Status 404 returned error can't find the container with id 3d210f796b41316414aef07d873143143d3d4307a526be8842f733ee633f04b0 Sep 10 04:42:29.380457 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.380390 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 04:42:29.384209 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.384107 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 04:42:29.386017 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.385279 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55d58df5db-tzss8"] Sep 10 04:42:29.396347 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:29.396310 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0c441d_8eca_4011_8c8f_c0c8b25750cd.slice/crio-a00c8c236560b9a373b469a44effe28e7c9c36dc36ddb7df3cff5def0ca77ac2 WatchSource:0}: Error finding container a00c8c236560b9a373b469a44effe28e7c9c36dc36ddb7df3cff5def0ca77ac2: Status 404 returned error can't find the container with id a00c8c236560b9a373b469a44effe28e7c9c36dc36ddb7df3cff5def0ca77ac2 Sep 10 04:42:29.401749 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.401401 2572 generic.go:356] "Generic (PLEG): container finished" podID="07f9db3c-da60-4f88-8327-61bd6936ecc3" containerID="36edfeb97b2ebaf4d0830bd7d5d4605899b39c45e8ee0ea29e5aeb7060233eaf" exitCode=0 Sep 10 04:42:29.401749 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.401546 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2lbl5" event={"ID":"07f9db3c-da60-4f88-8327-61bd6936ecc3","Type":"ContainerDied","Data":"36edfeb97b2ebaf4d0830bd7d5d4605899b39c45e8ee0ea29e5aeb7060233eaf"} Sep 10 04:42:29.411734 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.411581 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" event={"ID":"400d2682-da56-4ff8-adc3-0d514bb7f1be","Type":"ContainerStarted","Data":"f4d07998de9e52e11440b8fd65e91c3cb195b375c1de2293e70c9e683e81fd7a"} Sep 10 04:42:29.411734 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.411656 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" event={"ID":"400d2682-da56-4ff8-adc3-0d514bb7f1be","Type":"ContainerStarted","Data":"e80fb6bf85007e701c260909b0b4eef46c36e98092cad0ae8a139a9fb096c71b"} Sep 10 04:42:29.420094 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.420058 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-44wrh" event={"ID":"01e9fb97-7654-4ee0-88f3-baeef295cfbe","Type":"ContainerStarted","Data":"e5533bd613a88b5cce8616f2c2034201030c65e507c8b6553af34f26a0f3d93e"} Sep 10 04:42:29.420428 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.420368 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-44wrh" Sep 10 04:42:29.422323 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.422265 2572 patch_prober.go:28] interesting pod/downloads-85d98c998f-44wrh container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" start-of-body= Sep 10 04:42:29.423806 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.422968 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" event={"ID":"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a","Type":"ContainerStarted","Data":"88bcf464c75ec7665b102218591d34019f35bf76dcc513489875d7f55356a8a1"} Sep 10 04:42:29.423902 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.423844 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-44wrh" podUID="01e9fb97-7654-4ee0-88f3-baeef295cfbe" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" Sep 10 04:42:29.430823 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.430731 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" event={"ID":"ea31ddbd-8d97-440e-a793-a56f5fc9ab12","Type":"ContainerStarted","Data":"2f99536be5f8cbf8398c28a5191fbc38a1f3ea4116e4cd576405e4d841597342"} Sep 10 04:42:29.435145 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.435098 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" event={"ID":"2f193ed6-cb83-4049-8589-e5e8513606fe","Type":"ContainerStarted","Data":"ed0b0e0c07c2ebb495ba4421b72dd13269f13ad33f08df65486e1ff2b3ddf992"} Sep 10 04:42:29.437238 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.437197 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mh4x5"] Sep 10 04:42:29.439665 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.439614 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerStarted","Data":"6a68a71acc3c54642161646c54eaf44baf997c02add738155a9fcb086d9857ac"} Sep 10 04:42:29.441462 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.441426 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n2dx5"] Sep 10 04:42:29.444218 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.443981 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sgwg2" event={"ID":"34296475-0929-4b62-84ce-1666247486b3","Type":"ContainerStarted","Data":"531f7c12cd057b7fde1370d7d8c2e640db46e3a9d49055b396126d26d9dc9fa5"} Sep 10 04:42:29.450408 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:29.448441 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169267a0_c3db_4d30_b1ab_1e70511f19c6.slice/crio-3f84fab127286a685b05a9c026686f8774387b2283143293b30312bc9f63ae62 WatchSource:0}: Error finding container 3f84fab127286a685b05a9c026686f8774387b2283143293b30312bc9f63ae62: Status 404 returned error can't find the container with id 3f84fab127286a685b05a9c026686f8774387b2283143293b30312bc9f63ae62 Sep 10 04:42:29.450408 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.450041 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-44wrh" podStartSLOduration=1.933012645 podStartE2EDuration="19.450021124s" totalImagesPullingTime="17.517008479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:29.44884996 +0000 UTC m=+68.952683022" watchObservedRunningTime="2026-09-10 04:42:29.450021124 +0000 UTC m=+68.953854218" Sep 10 04:42:29.453476 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:29.453418 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcd4c9476-hzwws" event={"ID":"a89551a4-3391-4bde-a5df-c404fb2c1a6c","Type":"ContainerStarted","Data":"3d210f796b41316414aef07d873143143d3d4307a526be8842f733ee633f04b0"} Sep 10 04:42:29.455985 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:42:29.455961 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bf0532_232f_423a_add2_7c0ff0a5a9a9.slice/crio-a2c3ed4aa190ab8054ef75e4a404bf95606bee8512f1a2b02d7d07cc297e0072 WatchSource:0}: Error finding container a2c3ed4aa190ab8054ef75e4a404bf95606bee8512f1a2b02d7d07cc297e0072: Status 404 returned error can't find the container with id a2c3ed4aa190ab8054ef75e4a404bf95606bee8512f1a2b02d7d07cc297e0072 Sep 10 04:42:30.463837 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:30.463633 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n2dx5" event={"ID":"169267a0-c3db-4d30-b1ab-1e70511f19c6","Type":"ContainerStarted","Data":"3f84fab127286a685b05a9c026686f8774387b2283143293b30312bc9f63ae62"} Sep 10 04:42:30.470472 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:30.469975 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2lbl5" event={"ID":"07f9db3c-da60-4f88-8327-61bd6936ecc3","Type":"ContainerStarted","Data":"5c5274ff6f72676a8f06e6ca3b2cfdf264431c321094d79ac8f4124575ee5497"} Sep 10 04:42:30.470472 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:30.470047 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2lbl5" event={"ID":"07f9db3c-da60-4f88-8327-61bd6936ecc3","Type":"ContainerStarted","Data":"6fb2516acaca42f7c825acb8a24e90d3f72ebff5ad4db1f16a3c4496e074df3d"} Sep 10 04:42:30.477218 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:30.475775 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mh4x5" event={"ID":"46bf0532-232f-423a-add2-7c0ff0a5a9a9","Type":"ContainerStarted","Data":"a2c3ed4aa190ab8054ef75e4a404bf95606bee8512f1a2b02d7d07cc297e0072"} Sep 10 04:42:30.510305 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:30.509990 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" event={"ID":"400d2682-da56-4ff8-adc3-0d514bb7f1be","Type":"ContainerStarted","Data":"8b6529e22c729724108c33201b93c3984270c748fd384019e99e0fbd99f783f2"} Sep 10 04:42:30.531509 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:30.530576 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2lbl5" podStartSLOduration=2.5341610980000002 podStartE2EDuration="11.530548036s" totalImagesPullingTime="8.877770774s" totalInitContainerRuntime="118.616164ms" isStatefulPod=true podCreationTimestamp="2026-09-10 04:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:30.498466498 +0000 UTC m=+70.002299554" watchObservedRunningTime="2026-09-10 04:42:30.530548036 +0000 UTC m=+70.034381098" Sep 10 04:42:30.537759 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:30.537714 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" event={"ID":"d60fe704-87f9-4b56-8196-4354e0199fb3","Type":"ContainerStarted","Data":"cd45ba02a77e5826ef26c0a7385fc422589a5f544daee95c9a4613f1ab6e9439"} Sep 10 04:42:30.564530 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:30.563915 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-6t7c5" podStartSLOduration=2.926983012 podStartE2EDuration="11.563889433s" totalImagesPullingTime="8.636906421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:30.531313779 +0000 UTC m=+70.035146844" watchObservedRunningTime="2026-09-10 04:42:30.563889433 +0000 UTC m=+70.067722497" Sep 10 04:42:30.575152 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:30.575109 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerStarted","Data":"a00c8c236560b9a373b469a44effe28e7c9c36dc36ddb7df3cff5def0ca77ac2"} Sep 10 04:42:30.594221 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:30.594143 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-44wrh" Sep 10 04:42:30.616132 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:30.614981 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pgx4x" podStartSLOduration=2.6709380940000003 podStartE2EDuration="11.614959549s" totalImagesPullingTime="8.944021455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:30.565017523 +0000 UTC m=+70.068850585" watchObservedRunningTime="2026-09-10 04:42:30.614959549 +0000 UTC m=+70.118792611" Sep 10 04:42:40.349209 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:40.349169 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcd4c9476-hzwws"] Sep 10 04:42:41.646324 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:41.646277 2572 generic.go:356] "Generic (PLEG): container finished" podID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerID="61c0d84400b9356994eedc8257ca01378a49b1fc7384ad87004a46e61ab14f02" exitCode=0 Sep 10 04:42:41.646940 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:41.646841 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerDied","Data":"61c0d84400b9356994eedc8257ca01378a49b1fc7384ad87004a46e61ab14f02"} Sep 10 04:42:41.650487 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:41.650333 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" event={"ID":"ea31ddbd-8d97-440e-a793-a56f5fc9ab12","Type":"ContainerStarted","Data":"ebb65f0b1e76b08ffda6db1ed8a096c3861fbf2b8a551b7200fb61f6d6dc2ab1"} Sep 10 04:42:41.659792 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:41.659753 2572 generic.go:356] "Generic (PLEG): container finished" podID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerID="e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1" exitCode=0 Sep 10 04:42:41.659957 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:41.659889 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerDied","Data":"e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1"} Sep 10 04:42:41.669264 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:41.668883 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n2dx5" event={"ID":"169267a0-c3db-4d30-b1ab-1e70511f19c6","Type":"ContainerStarted","Data":"2e26976fac152f6f1e1ade6dd22089fe12c6f5df8459d2b977d2b6dd66820c1d"} Sep 10 04:42:41.775717 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:41.775626 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" podStartSLOduration=5.927029414 podStartE2EDuration="17.775605067s" totalImagesPullingTime="11.848575653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:41.775191888 +0000 UTC m=+81.279024950" watchObservedRunningTime="2026-09-10 04:42:41.775605067 +0000 UTC m=+81.279438176" Sep 10 04:42:42.675883 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.675793 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sgwg2" event={"ID":"34296475-0929-4b62-84ce-1666247486b3","Type":"ContainerStarted","Data":"0e99bfe1e3460ad9d2264cc1655e1ce17ac22eb5b2eff199d4949f16edb6bea4"} Sep 10 04:42:42.676347 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.675907 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:42:42.678816 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.678653 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mh4x5" event={"ID":"46bf0532-232f-423a-add2-7c0ff0a5a9a9","Type":"ContainerStarted","Data":"4c41b2aa7a602dd100fc00e6b0187ce0ff4e0af3a1d810aa4bbbcdcafe747ba7"} Sep 10 04:42:42.680780 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.680749 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcd4c9476-hzwws" event={"ID":"a89551a4-3391-4bde-a5df-c404fb2c1a6c","Type":"ContainerStarted","Data":"cc9d663af701361f947e741b9c9c39d058c2e3e057e6e11f58803ef96336b4e9"} Sep 10 04:42:42.683634 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.683597 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n2dx5" event={"ID":"169267a0-c3db-4d30-b1ab-1e70511f19c6","Type":"ContainerStarted","Data":"05ef38109b98b3c941387db7fb01b55742936279f87c75b186442d8b20f0a9f2"} Sep 10 04:42:42.686033 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.685997 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" event={"ID":"2f193ed6-cb83-4049-8589-e5e8513606fe","Type":"ContainerStarted","Data":"6c3e7b8165944985f66273bb101dc04fb70971347d7d78a1675561a48caf1f8e"} Sep 10 04:42:42.686345 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.686324 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" Sep 10 04:42:42.689523 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.689297 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" event={"ID":"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a","Type":"ContainerStarted","Data":"3c17416f986ebb63748376d71aa61c7464e001cf44435edf979bcbb9fbf428cd"} Sep 10 04:42:42.689523 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.689394 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" event={"ID":"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a","Type":"ContainerStarted","Data":"f43a3422171d15721c24979c93e5ec61ef5ce5a9d13e99aea7aee4e6c6c87f19"} Sep 10 04:42:42.689523 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.689436 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" event={"ID":"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a","Type":"ContainerStarted","Data":"8c25ab94d9c7f5f3f4edfd6dddbed450e7f45834f6a18f6952ebf2e774de0886"} Sep 10 04:42:42.691925 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.691879 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" Sep 10 04:42:42.694960 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.694909 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sgwg2" podStartSLOduration=69.791218534 podStartE2EDuration="1m21.694891974s" totalImagesPullingTime="11.90367344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:42.692482417 +0000 UTC m=+82.196315479" watchObservedRunningTime="2026-09-10 04:42:42.694891974 +0000 UTC m=+82.198725034" Sep 10 04:42:42.719010 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.718934 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n2dx5" podStartSLOduration=71.979409332 podStartE2EDuration="1m21.718909175s" totalImagesPullingTime="9.739499843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:42.718213322 +0000 UTC m=+82.222046399" watchObservedRunningTime="2026-09-10 04:42:42.718909175 +0000 UTC m=+82.222742237" Sep 10 04:42:42.738051 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.737984 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dcd4c9476-hzwws" podStartSLOduration=9.855295758 podStartE2EDuration="21.737959608s" totalImagesPullingTime="11.88266385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:42.736837549 +0000 UTC m=+82.240670612" watchObservedRunningTime="2026-09-10 04:42:42.737959608 +0000 UTC m=+82.241792670" Sep 10 04:42:42.758093 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.758026 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mh4x5" podStartSLOduration=69.603002236 podStartE2EDuration="1m21.758004396s" totalImagesPullingTime="12.15500216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 04:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:42.753285704 +0000 UTC m=+82.257118763" watchObservedRunningTime="2026-09-10 04:42:42.758004396 +0000 UTC m=+82.261837459" Sep 10 04:42:42.772789 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:42.772559 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q9fws" podStartSLOduration=6.823370431 podStartE2EDuration="18.772532387s" totalImagesPullingTime="11.949161956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:42.770682994 +0000 UTC m=+82.274516060" watchObservedRunningTime="2026-09-10 04:42:42.772532387 +0000 UTC m=+82.276365453" Sep 10 04:42:44.516980 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:44.516935 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:44.517604 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:44.517053 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:42:44.702112 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:44.702070 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" event={"ID":"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a","Type":"ContainerStarted","Data":"695ffe1cd1c36c757517905dd4d24c8369ef16b5a7faf908d7839b65a661a6ad"} Sep 10 04:42:44.704400 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:44.704351 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerStarted","Data":"c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39"} Sep 10 04:42:45.717819 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:45.716619 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" event={"ID":"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a","Type":"ContainerStarted","Data":"692c1d81bc23fbc71cbea950e33506c58ef4f641d4f5b8719c2760c0f15a3acc"} Sep 10 04:42:45.717819 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:45.716713 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" event={"ID":"7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a","Type":"ContainerStarted","Data":"39c99496f77313cae0ecae8f76420bf60820a81054db431c2c68aa2970b8c3b5"} Sep 10 04:42:45.717819 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:45.717489 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:45.721507 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:45.721471 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerStarted","Data":"eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d"} Sep 10 04:42:45.737436 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:45.737181 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerStarted","Data":"5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5"} Sep 10 04:42:45.737436 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:45.737279 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerStarted","Data":"c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c"} Sep 10 04:42:45.737436 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:45.737325 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerStarted","Data":"d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7"} Sep 10 04:42:45.737436 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:45.737363 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerStarted","Data":"43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10"} Sep 10 04:42:45.778356 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:45.778138 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" podStartSLOduration=10.229473191 podStartE2EDuration="23.778117301s" totalImagesPullingTime="13.54864411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:45.773782142 +0000 UTC m=+85.277615238" watchObservedRunningTime="2026-09-10 04:42:45.778117301 +0000 UTC m=+85.281950362" Sep 10 04:42:46.745089 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:46.745046 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerStarted","Data":"b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187"} Sep 10 04:42:46.748817 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:46.748784 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerStarted","Data":"a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80"} Sep 10 04:42:46.748997 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:46.748849 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerStarted","Data":"e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb"} Sep 10 04:42:46.748997 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:46.748877 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerStarted","Data":"bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d"} Sep 10 04:42:46.748997 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:46.748900 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerStarted","Data":"5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e"} Sep 10 04:42:46.748997 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:46.748923 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerStarted","Data":"a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146"} Sep 10 04:42:46.756924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:46.756891 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55d58df5db-tzss8" Sep 10 04:42:46.775372 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:46.775315 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=12.91968148 podStartE2EDuration="26.775297937s" totalImagesPullingTime="13.842864874s" totalInitContainerRuntime="12.751583ms" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:46.772477411 +0000 UTC m=+86.276310494" watchObservedRunningTime="2026-09-10 04:42:46.775297937 +0000 UTC m=+86.279130999" Sep 10 04:42:46.828198 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:46.828134 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.165761962 podStartE2EDuration="21.828111791s" totalImagesPullingTime="15.639246221s" totalInitContainerRuntime="23.103608ms" isStatefulPod=false podCreationTimestamp="2026-09-10 04:42:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:42:46.825899551 +0000 UTC m=+86.329732745" watchObservedRunningTime="2026-09-10 04:42:46.828111791 +0000 UTC m=+86.331944874" Sep 10 04:42:51.212107 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:51.212057 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:42:51.479210 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:42:51.479126 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:43:04.522698 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:04.522649 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:43:04.526594 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:04.526557 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-77b8b9cc78-d2hfw" Sep 10 04:43:07.710862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:07.710796 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dcd4c9476-hzwws" podUID="a89551a4-3391-4bde-a5df-c404fb2c1a6c" containerName="console" containerID="cri-o://cc9d663af701361f947e741b9c9c39d058c2e3e057e6e11f58803ef96336b4e9" gracePeriod=15 Sep 10 04:43:07.846378 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:07.846340 2572 generic.go:356] "Generic (PLEG): container finished" podID="a89551a4-3391-4bde-a5df-c404fb2c1a6c" containerID="cc9d663af701361f947e741b9c9c39d058c2e3e057e6e11f58803ef96336b4e9" exitCode=0 Sep 10 04:43:07.846557 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:07.846408 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcd4c9476-hzwws" event={"ID":"a89551a4-3391-4bde-a5df-c404fb2c1a6c","Type":"ContainerDied","Data":"cc9d663af701361f947e741b9c9c39d058c2e3e057e6e11f58803ef96336b4e9"} Sep 10 04:43:07.962036 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:07.961958 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:43:08.051977 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.051936 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjwjj\" (UniqueName: \"kubernetes.io/projected/a89551a4-3391-4bde-a5df-c404fb2c1a6c-kube-api-access-cjwjj\") pod \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " Sep 10 04:43:08.052159 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.051999 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-oauth-config\") pod \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " Sep 10 04:43:08.052159 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.052063 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-service-ca\") pod \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " Sep 10 04:43:08.052159 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.052115 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-oauth-serving-cert\") pod \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " Sep 10 04:43:08.052159 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.052150 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-config\") pod \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " Sep 10 04:43:08.052380 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.052188 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-serving-cert\") pod \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\" (UID: \"a89551a4-3391-4bde-a5df-c404fb2c1a6c\") " Sep 10 04:43:08.052529 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.052497 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-service-ca" (OuterVolumeSpecName: "service-ca") pod "a89551a4-3391-4bde-a5df-c404fb2c1a6c" (UID: "a89551a4-3391-4bde-a5df-c404fb2c1a6c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:08.052529 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.052518 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a89551a4-3391-4bde-a5df-c404fb2c1a6c" (UID: "a89551a4-3391-4bde-a5df-c404fb2c1a6c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:08.052673 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.052601 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-config" (OuterVolumeSpecName: "console-config") pod "a89551a4-3391-4bde-a5df-c404fb2c1a6c" (UID: "a89551a4-3391-4bde-a5df-c404fb2c1a6c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:08.054653 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.054583 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a89551a4-3391-4bde-a5df-c404fb2c1a6c" (UID: "a89551a4-3391-4bde-a5df-c404fb2c1a6c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:08.054765 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.054655 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a89551a4-3391-4bde-a5df-c404fb2c1a6c" (UID: "a89551a4-3391-4bde-a5df-c404fb2c1a6c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:08.054765 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.054707 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89551a4-3391-4bde-a5df-c404fb2c1a6c-kube-api-access-cjwjj" (OuterVolumeSpecName: "kube-api-access-cjwjj") pod "a89551a4-3391-4bde-a5df-c404fb2c1a6c" (UID: "a89551a4-3391-4bde-a5df-c404fb2c1a6c"). InnerVolumeSpecName "kube-api-access-cjwjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:43:08.153830 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.153791 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-oauth-config\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:08.153830 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.153824 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-service-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:08.153830 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.153834 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-oauth-serving-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:08.153830 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.153843 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-config\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:08.154098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.153853 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a89551a4-3391-4bde-a5df-c404fb2c1a6c-console-serving-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:08.154098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.153862 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjwjj\" (UniqueName: \"kubernetes.io/projected/a89551a4-3391-4bde-a5df-c404fb2c1a6c-kube-api-access-cjwjj\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:08.852180 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.852147 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcd4c9476-hzwws" Sep 10 04:43:08.852180 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.852156 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcd4c9476-hzwws" event={"ID":"a89551a4-3391-4bde-a5df-c404fb2c1a6c","Type":"ContainerDied","Data":"3d210f796b41316414aef07d873143143d3d4307a526be8842f733ee633f04b0"} Sep 10 04:43:08.852858 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.852212 2572 scope.go:117] "RemoveContainer" containerID="cc9d663af701361f947e741b9c9c39d058c2e3e057e6e11f58803ef96336b4e9" Sep 10 04:43:08.874007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.873971 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcd4c9476-hzwws"] Sep 10 04:43:08.880340 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:08.880307 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dcd4c9476-hzwws"] Sep 10 04:43:09.036011 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:09.035975 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a89551a4-3391-4bde-a5df-c404fb2c1a6c" path="/var/lib/kubelet/pods/a89551a4-3391-4bde-a5df-c404fb2c1a6c/volumes" Sep 10 04:43:13.703512 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:13.703477 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sgwg2" Sep 10 04:43:26.212422 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:26.212366 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:26.272205 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:26.272172 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:26.951230 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:26.951201 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:39.914661 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:39.914625 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 04:43:39.915157 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:39.914932 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="alertmanager" containerID="cri-o://c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39" gracePeriod=120 Sep 10 04:43:39.915157 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:39.915011 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy-web" containerID="cri-o://d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7" gracePeriod=120 Sep 10 04:43:39.915279 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:39.915041 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy-metric" containerID="cri-o://5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5" gracePeriod=120 Sep 10 04:43:39.915279 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:39.915041 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy" containerID="cri-o://c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c" gracePeriod=120 Sep 10 04:43:39.915279 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:39.915094 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="config-reloader" containerID="cri-o://43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10" gracePeriod=120 Sep 10 04:43:39.915422 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:39.915090 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="prom-label-proxy" containerID="cri-o://b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187" gracePeriod=120 Sep 10 04:43:40.995893 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:40.995861 2572 generic.go:356] "Generic (PLEG): container finished" podID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerID="b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187" exitCode=0 Sep 10 04:43:40.995893 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:40.995887 2572 generic.go:356] "Generic (PLEG): container finished" podID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerID="c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c" exitCode=0 Sep 10 04:43:40.995893 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:40.995894 2572 generic.go:356] "Generic (PLEG): container finished" podID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerID="43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10" exitCode=0 Sep 10 04:43:40.995893 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:40.995901 2572 generic.go:356] "Generic (PLEG): container finished" podID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerID="c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39" exitCode=0 Sep 10 04:43:40.996384 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:40.995929 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerDied","Data":"b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187"} Sep 10 04:43:40.996384 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:40.995992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerDied","Data":"c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c"} Sep 10 04:43:40.996384 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:40.996019 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerDied","Data":"43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10"} Sep 10 04:43:40.996384 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:40.996044 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerDied","Data":"c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39"} Sep 10 04:43:41.170037 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.170007 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:41.254758 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.254663 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.254758 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.254723 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-tls-assets\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.254977 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.254885 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-out\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.254977 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.254955 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-trusted-ca-bundle\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.255084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.254998 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-main-tls\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.255084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.255034 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-web\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.255084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.255062 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-cluster-tls-config\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.255233 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.255096 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dbgd\" (UniqueName: \"kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-kube-api-access-5dbgd\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.255233 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.255135 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-volume\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.255233 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.255169 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-metrics-client-ca\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.255233 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.255200 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.255417 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.255232 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-main-db\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.255417 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.255285 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-web-config\") pod \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\" (UID: \"0d0c441d-8eca-4011-8c8f-c0c8b25750cd\") " Sep 10 04:43:41.255417 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.255384 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:41.255610 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.255593 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.256908 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.256876 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:41.257046 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.257011 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:43:41.258440 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.258380 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:41.258578 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.258534 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:41.258815 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.258761 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:43:41.258935 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.258903 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-out" (OuterVolumeSpecName: "config-out") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:43:41.259684 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.259649 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:41.259814 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.259787 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-volume" (OuterVolumeSpecName: "config-volume") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:41.260164 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.260137 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:41.260535 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.260511 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-kube-api-access-5dbgd" (OuterVolumeSpecName: "kube-api-access-5dbgd") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "kube-api-access-5dbgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:43:41.264082 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.264055 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:41.271293 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.271258 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-web-config" (OuterVolumeSpecName: "web-config") pod "0d0c441d-8eca-4011-8c8f-c0c8b25750cd" (UID: "0d0c441d-8eca-4011-8c8f-c0c8b25750cd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:41.355910 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355872 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.355910 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355901 2572 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-cluster-tls-config\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.355910 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355911 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dbgd\" (UniqueName: \"kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-kube-api-access-5dbgd\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.355910 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355922 2572 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.356181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355931 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-metrics-client-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.356181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355940 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.356181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355948 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-alertmanager-main-db\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.356181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355956 2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-web-config\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.356181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355964 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.356181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355973 2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-tls-assets\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.356181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355983 2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-config-out\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:41.356181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:41.355991 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d0c441d-8eca-4011-8c8f-c0c8b25750cd-secret-alertmanager-main-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:42.004270 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.004232 2572 generic.go:356] "Generic (PLEG): container finished" podID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerID="5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5" exitCode=0 Sep 10 04:43:42.004270 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.004264 2572 generic.go:356] "Generic (PLEG): container finished" podID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerID="d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7" exitCode=0 Sep 10 04:43:42.004736 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.004292 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerDied","Data":"5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5"} Sep 10 04:43:42.004736 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.004320 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.004736 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.004390 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerDied","Data":"d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7"} Sep 10 04:43:42.004736 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.004424 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d0c441d-8eca-4011-8c8f-c0c8b25750cd","Type":"ContainerDied","Data":"a00c8c236560b9a373b469a44effe28e7c9c36dc36ddb7df3cff5def0ca77ac2"} Sep 10 04:43:42.004736 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.004475 2572 scope.go:117] "RemoveContainer" containerID="b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187" Sep 10 04:43:42.014181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.014153 2572 scope.go:117] "RemoveContainer" containerID="5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5" Sep 10 04:43:42.021961 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.021941 2572 scope.go:117] "RemoveContainer" containerID="c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c" Sep 10 04:43:42.030367 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.030240 2572 scope.go:117] "RemoveContainer" containerID="d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7" Sep 10 04:43:42.031642 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.031620 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 04:43:42.036103 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.036080 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 04:43:42.039398 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.039373 2572 scope.go:117] "RemoveContainer" containerID="43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10" Sep 10 04:43:42.046918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.046897 2572 scope.go:117] "RemoveContainer" containerID="c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39" Sep 10 04:43:42.054695 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.054670 2572 scope.go:117] "RemoveContainer" containerID="e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1" Sep 10 04:43:42.062971 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.062945 2572 scope.go:117] "RemoveContainer" containerID="b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187" Sep 10 04:43:42.063354 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:42.063330 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187\": container with ID starting with b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187 not found: ID does not exist" containerID="b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187" Sep 10 04:43:42.063427 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.063363 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187"} err="failed to get container status \"b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187\": rpc error: code = NotFound desc = could not find container \"b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187\": container with ID starting with b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187 not found: ID does not exist" Sep 10 04:43:42.063427 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.063406 2572 scope.go:117] "RemoveContainer" containerID="5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5" Sep 10 04:43:42.063722 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:42.063701 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5\": container with ID starting with 5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5 not found: ID does not exist" containerID="5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5" Sep 10 04:43:42.063797 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.063732 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5"} err="failed to get container status \"5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5\": rpc error: code = NotFound desc = could not find container \"5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5\": container with ID starting with 5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5 not found: ID does not exist" Sep 10 04:43:42.063797 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.063756 2572 scope.go:117] "RemoveContainer" containerID="c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c" Sep 10 04:43:42.064029 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:42.064014 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c\": container with ID starting with c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c not found: ID does not exist" containerID="c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c" Sep 10 04:43:42.064089 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.064038 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c"} err="failed to get container status \"c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c\": rpc error: code = NotFound desc = could not find container \"c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c\": container with ID starting with c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c not found: ID does not exist" Sep 10 04:43:42.064089 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.064059 2572 scope.go:117] "RemoveContainer" containerID="d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7" Sep 10 04:43:42.064340 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:42.064317 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7\": container with ID starting with d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7 not found: ID does not exist" containerID="d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7" Sep 10 04:43:42.064421 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.064343 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7"} err="failed to get container status \"d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7\": rpc error: code = NotFound desc = could not find container \"d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7\": container with ID starting with d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7 not found: ID does not exist" Sep 10 04:43:42.064421 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.064358 2572 scope.go:117] "RemoveContainer" containerID="43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10" Sep 10 04:43:42.064661 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:42.064643 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10\": container with ID starting with 43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10 not found: ID does not exist" containerID="43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10" Sep 10 04:43:42.064717 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.064664 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10"} err="failed to get container status \"43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10\": rpc error: code = NotFound desc = could not find container \"43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10\": container with ID starting with 43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10 not found: ID does not exist" Sep 10 04:43:42.064717 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.064678 2572 scope.go:117] "RemoveContainer" containerID="c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39" Sep 10 04:43:42.064943 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:42.064922 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39\": container with ID starting with c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39 not found: ID does not exist" containerID="c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39" Sep 10 04:43:42.064992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.064951 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39"} err="failed to get container status \"c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39\": rpc error: code = NotFound desc = could not find container \"c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39\": container with ID starting with c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39 not found: ID does not exist" Sep 10 04:43:42.064992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.064970 2572 scope.go:117] "RemoveContainer" containerID="e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1" Sep 10 04:43:42.065219 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:42.065199 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1\": container with ID starting with e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1 not found: ID does not exist" containerID="e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1" Sep 10 04:43:42.065315 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.065221 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1"} err="failed to get container status \"e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1\": rpc error: code = NotFound desc = could not find container \"e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1\": container with ID starting with e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1 not found: ID does not exist" Sep 10 04:43:42.065315 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.065236 2572 scope.go:117] "RemoveContainer" containerID="b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187" Sep 10 04:43:42.065541 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.065518 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187"} err="failed to get container status \"b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187\": rpc error: code = NotFound desc = could not find container \"b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187\": container with ID starting with b08b6df72da63e3dea36aad6dcbeff1428427e2dd396ddf31661b0b1d17d8187 not found: ID does not exist" Sep 10 04:43:42.065593 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.065544 2572 scope.go:117] "RemoveContainer" containerID="5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5" Sep 10 04:43:42.065803 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.065784 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5"} err="failed to get container status \"5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5\": rpc error: code = NotFound desc = could not find container \"5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5\": container with ID starting with 5908f398621f89ce0d91ef8e06926f69e4d2e9a9c58d5bd12df8f5186fa5c1e5 not found: ID does not exist" Sep 10 04:43:42.065803 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.065803 2572 scope.go:117] "RemoveContainer" containerID="c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c" Sep 10 04:43:42.066070 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.066046 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c"} err="failed to get container status \"c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c\": rpc error: code = NotFound desc = could not find container \"c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c\": container with ID starting with c21f8f18f03d97b5e31ccc20faadcc74b5d42a5a00613f37f567aeac3ddd672c not found: ID does not exist" Sep 10 04:43:42.066148 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.066072 2572 scope.go:117] "RemoveContainer" containerID="d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7" Sep 10 04:43:42.066337 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.066318 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7"} err="failed to get container status \"d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7\": rpc error: code = NotFound desc = could not find container \"d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7\": container with ID starting with d46c96b74806e6f13b32b8bfaba3265386c96234da7c7201b0f7bbe99bedd4f7 not found: ID does not exist" Sep 10 04:43:42.066337 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.066337 2572 scope.go:117] "RemoveContainer" containerID="43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10" Sep 10 04:43:42.066731 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.066705 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10"} err="failed to get container status \"43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10\": rpc error: code = NotFound desc = could not find container \"43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10\": container with ID starting with 43a5a2415cf1862153b3edce3fecd5247111403a6b3f1d6fcdd4603add57ab10 not found: ID does not exist" Sep 10 04:43:42.066839 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.066734 2572 scope.go:117] "RemoveContainer" containerID="c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39" Sep 10 04:43:42.067256 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.067208 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39"} err="failed to get container status \"c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39\": rpc error: code = NotFound desc = could not find container \"c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39\": container with ID starting with c24031aa6a21d184f16b7c34a7cad5417d99ce396733552d5e0a4ef1e1eebc39 not found: ID does not exist" Sep 10 04:43:42.067256 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.067244 2572 scope.go:117] "RemoveContainer" containerID="e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1" Sep 10 04:43:42.067602 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.067574 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1"} err="failed to get container status \"e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1\": rpc error: code = NotFound desc = could not find container \"e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1\": container with ID starting with e1c3d917e49725bdbb653246924305257884c12a4d54f26d0b4c438dbdce92e1 not found: ID does not exist" Sep 10 04:43:42.069104 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069083 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 04:43:42.069404 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069389 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="config-reloader" Sep 10 04:43:42.069404 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069405 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="config-reloader" Sep 10 04:43:42.069543 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069414 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy-web" Sep 10 04:43:42.069543 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069421 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy-web" Sep 10 04:43:42.069543 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069427 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy" Sep 10 04:43:42.069543 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069432 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy" Sep 10 04:43:42.069543 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069489 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="prom-label-proxy" Sep 10 04:43:42.069543 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069498 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="prom-label-proxy" Sep 10 04:43:42.069543 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069510 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="alertmanager" Sep 10 04:43:42.069543 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069518 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="alertmanager" Sep 10 04:43:42.069543 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069534 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="init-config-reloader" Sep 10 04:43:42.069543 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069543 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="init-config-reloader" Sep 10 04:43:42.069992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069556 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a89551a4-3391-4bde-a5df-c404fb2c1a6c" containerName="console" Sep 10 04:43:42.069992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069564 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a89551a4-3391-4bde-a5df-c404fb2c1a6c" containerName="console" Sep 10 04:43:42.069992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069573 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy-metric" Sep 10 04:43:42.069992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069581 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy-metric" Sep 10 04:43:42.069992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069637 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="prom-label-proxy" Sep 10 04:43:42.069992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069645 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy-metric" Sep 10 04:43:42.069992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069654 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="config-reloader" Sep 10 04:43:42.069992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069660 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a89551a4-3391-4bde-a5df-c404fb2c1a6c" containerName="console" Sep 10 04:43:42.069992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069665 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy" Sep 10 04:43:42.069992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069670 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="alertmanager" Sep 10 04:43:42.069992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.069677 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" containerName="kube-rbac-proxy-web" Sep 10 04:43:42.075083 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.075059 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.077908 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.077733 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 04:43:42.077908 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.077733 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 04:43:42.077908 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.077777 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 04:43:42.077908 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.077785 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 04:43:42.077908 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.077808 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 04:43:42.077908 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.077734 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 04:43:42.077908 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.077881 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 04:43:42.078274 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.078184 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 04:43:42.078274 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.078222 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jxftf\"" Sep 10 04:43:42.082968 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.082948 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 04:43:42.088725 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.088696 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 04:43:42.159906 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.159868 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.159906 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.159912 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.160114 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.159972 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.160114 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.160019 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.160114 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.160051 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-config-out\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.160114 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.160067 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.160114 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.160099 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.160320 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.160131 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.160320 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.160183 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4m9x\" (UniqueName: \"kubernetes.io/projected/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-kube-api-access-t4m9x\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.160320 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.160213 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.160320 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.160244 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-web-config\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.160320 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.160260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.160320 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.160275 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.260832 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.260738 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.260832 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.260774 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.260832 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.260795 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.260832 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.260827 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-config-out\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.261151 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.260853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.261151 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.260880 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.261151 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.260961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.261151 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.261001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4m9x\" (UniqueName: \"kubernetes.io/projected/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-kube-api-access-t4m9x\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.261151 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.261025 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.261151 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.261055 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-web-config\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.261151 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.261082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.261151 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.261106 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.261151 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.261152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.261730 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.261706 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.262016 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.261991 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.262530 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.262416 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.263888 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.263848 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.264285 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.264166 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.264285 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.264232 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-config-out\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.264975 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.264639 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-web-config\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.264975 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.264748 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.264975 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.264832 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.264975 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.264931 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.265168 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.265028 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.265610 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.265592 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.269680 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.269656 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4m9x\" (UniqueName: \"kubernetes.io/projected/8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f-kube-api-access-t4m9x\") pod \"alertmanager-main-0\" (UID: \"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.385043 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.385002 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 04:43:42.666112 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:42.666080 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 04:43:42.668619 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:43:42.668591 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a51d9fd_0ce4_4c0a_ac56_14f6b6d2537f.slice/crio-8064ea23aa5da148475df68d760cfbc987164ff62338e87fa1efea758fce396c WatchSource:0}: Error finding container 8064ea23aa5da148475df68d760cfbc987164ff62338e87fa1efea758fce396c: Status 404 returned error can't find the container with id 8064ea23aa5da148475df68d760cfbc987164ff62338e87fa1efea758fce396c Sep 10 04:43:43.009854 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:43.009820 2572 generic.go:356] "Generic (PLEG): container finished" podID="8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f" containerID="b80ea9c5a936c3be53f4da6875e9f75bf3f094bcba5da5bf5e45993cd4276486" exitCode=0 Sep 10 04:43:43.010272 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:43.009919 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f","Type":"ContainerDied","Data":"b80ea9c5a936c3be53f4da6875e9f75bf3f094bcba5da5bf5e45993cd4276486"} Sep 10 04:43:43.010272 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:43.010003 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f","Type":"ContainerStarted","Data":"8064ea23aa5da148475df68d760cfbc987164ff62338e87fa1efea758fce396c"} Sep 10 04:43:43.037842 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:43.037807 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d0c441d-8eca-4011-8c8f-c0c8b25750cd" path="/var/lib/kubelet/pods/0d0c441d-8eca-4011-8c8f-c0c8b25750cd/volumes" Sep 10 04:43:44.019649 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.019614 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f","Type":"ContainerStarted","Data":"b4a841e2acebc85d480c672d8ad0a0b3e004d169c82b1c266040bdcc1fcd1b08"} Sep 10 04:43:44.020000 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.019716 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f","Type":"ContainerStarted","Data":"df25f05d0a90afc718bcdf9559358b56e9e35359a645387cb2a4f15a6b3d1787"} Sep 10 04:43:44.020000 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.019744 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f","Type":"ContainerStarted","Data":"e37a4e64e534f8a3f67cae1be8b415413d36b2f5373157f970a74c3a6217f4d6"} Sep 10 04:43:44.020000 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.019768 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f","Type":"ContainerStarted","Data":"f644dbece998f11afe87876fc41e3a7a5058da483f63197948deb61094d16632"} Sep 10 04:43:44.020000 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.019791 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f","Type":"ContainerStarted","Data":"b8db89d0a21c2779f2ce6946ef86a382ee7da1fa0bc12ff371c59381efd196cf"} Sep 10 04:43:44.020000 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.019813 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f","Type":"ContainerStarted","Data":"2309e92e29bf5b559a10c038111472de40260885c7b774eb143fa3ac43b342fa"} Sep 10 04:43:44.046585 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.046516 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.030813728 podStartE2EDuration="2.046496223s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.682495ms" isStatefulPod=false podCreationTimestamp="2026-09-10 04:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:43:44.04503287 +0000 UTC m=+143.548865929" watchObservedRunningTime="2026-09-10 04:43:44.046496223 +0000 UTC m=+143.550329283" Sep 10 04:43:44.252154 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.252111 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 04:43:44.252877 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.252509 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="prometheus" containerID="cri-o://eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d" gracePeriod=600 Sep 10 04:43:44.252877 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.252555 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80" gracePeriod=600 Sep 10 04:43:44.252877 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.252593 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy" containerID="cri-o://e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb" gracePeriod=600 Sep 10 04:43:44.252877 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.252558 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy-web" containerID="cri-o://bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d" gracePeriod=600 Sep 10 04:43:44.252877 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.252706 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="thanos-sidecar" containerID="cri-o://5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e" gracePeriod=600 Sep 10 04:43:44.252877 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:44.252767 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="config-reloader" containerID="cri-o://a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146" gracePeriod=600 Sep 10 04:43:45.028185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.028145 2572 generic.go:356] "Generic (PLEG): container finished" podID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerID="a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80" exitCode=0 Sep 10 04:43:45.028185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.028173 2572 generic.go:356] "Generic (PLEG): container finished" podID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerID="e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb" exitCode=0 Sep 10 04:43:45.028185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.028183 2572 generic.go:356] "Generic (PLEG): container finished" podID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerID="5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e" exitCode=0 Sep 10 04:43:45.028185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.028190 2572 generic.go:356] "Generic (PLEG): container finished" podID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerID="a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146" exitCode=0 Sep 10 04:43:45.028185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.028197 2572 generic.go:356] "Generic (PLEG): container finished" podID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerID="eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d" exitCode=0 Sep 10 04:43:45.028864 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.028216 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerDied","Data":"a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80"} Sep 10 04:43:45.028864 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.028278 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerDied","Data":"e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb"} Sep 10 04:43:45.028864 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.028306 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerDied","Data":"5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e"} Sep 10 04:43:45.028864 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.028329 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerDied","Data":"a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146"} Sep 10 04:43:45.028864 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.028352 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerDied","Data":"eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d"} Sep 10 04:43:45.505040 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.505005 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:45.595590 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595491 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-kube-rbac-proxy\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595590 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595534 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-thanos-prometheus-http-client-file\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595590 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595559 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-config-out\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595590 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595576 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-metrics-client-ca\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595595 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-serving-certs-ca-bundle\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595615 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-tls-assets\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595640 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-2\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595662 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-1\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595687 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-web-config\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595718 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-grpc-tls\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595747 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-db\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595781 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vf4h\" (UniqueName: \"kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-kube-api-access-4vf4h\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595835 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-tls\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595865 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-kubelet-serving-ca-bundle\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.595924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595896 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.596544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595950 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-config\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.596544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595977 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-metrics-client-certs\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.596544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.595967 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:45.596544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.596000 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-0\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.596544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.596015 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:45.596544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.596041 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.596544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.596073 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-trusted-ca-bundle\") pod \"41bcb40e-ba1d-4522-8428-4e811b2717fe\" (UID: \"41bcb40e-ba1d-4522-8428-4e811b2717fe\") " Sep 10 04:43:45.596544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.596385 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-metrics-client-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.596544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.596404 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.597022 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.596785 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:45.598715 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.598294 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:45.598715 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.598340 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:45.598715 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.598654 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:45.599015 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.598975 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:45.599511 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.599140 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-config-out" (OuterVolumeSpecName: "config-out") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:43:45.599511 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.599147 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:45.599932 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.599893 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-kube-api-access-4vf4h" (OuterVolumeSpecName: "kube-api-access-4vf4h") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "kube-api-access-4vf4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:43:45.600237 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.600211 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:43:45.600332 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.600290 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:43:45.600498 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.600436 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:45.600872 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.600848 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-config" (OuterVolumeSpecName: "config") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:45.601009 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.600958 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:45.601196 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.601176 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:45.601715 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.601694 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:45.601792 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.601755 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:43:45.602159 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.602134 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:45.610279 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.610241 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-web-config" (OuterVolumeSpecName: "web-config") pod "41bcb40e-ba1d-4522-8428-4e811b2717fe" (UID: "41bcb40e-ba1d-4522-8428-4e811b2717fe"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:43:45.697498 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697423 2572 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-config\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697498 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697491 2572 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-metrics-client-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697498 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697504 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697520 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697534 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697546 2572 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-kube-rbac-proxy\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697560 2572 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697572 2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-config-out\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697583 2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-tls-assets\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697594 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697607 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697619 2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-web-config\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697630 2572 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-grpc-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697643 2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/41bcb40e-ba1d-4522-8428-4e811b2717fe-prometheus-k8s-db\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697654 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vf4h\" (UniqueName: \"kubernetes.io/projected/41bcb40e-ba1d-4522-8428-4e811b2717fe-kube-api-access-4vf4h\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697667 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697679 2572 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41bcb40e-ba1d-4522-8428-4e811b2717fe-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:45.697781 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:45.697691 2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/41bcb40e-ba1d-4522-8428-4e811b2717fe-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:43:46.036094 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.036061 2572 generic.go:356] "Generic (PLEG): container finished" podID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerID="bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d" exitCode=0 Sep 10 04:43:46.036525 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.036129 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerDied","Data":"bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d"} Sep 10 04:43:46.036525 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.036134 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.036525 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.036185 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"41bcb40e-ba1d-4522-8428-4e811b2717fe","Type":"ContainerDied","Data":"6a68a71acc3c54642161646c54eaf44baf997c02add738155a9fcb086d9857ac"} Sep 10 04:43:46.036525 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.036216 2572 scope.go:117] "RemoveContainer" containerID="a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80" Sep 10 04:43:46.045232 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.045204 2572 scope.go:117] "RemoveContainer" containerID="e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb" Sep 10 04:43:46.053088 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.053067 2572 scope.go:117] "RemoveContainer" containerID="bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d" Sep 10 04:43:46.062146 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.062079 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 04:43:46.064471 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.064429 2572 scope.go:117] "RemoveContainer" containerID="5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e" Sep 10 04:43:46.068623 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.068596 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 04:43:46.072937 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.072915 2572 scope.go:117] "RemoveContainer" containerID="a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146" Sep 10 04:43:46.080937 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.080917 2572 scope.go:117] "RemoveContainer" containerID="eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d" Sep 10 04:43:46.089141 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.089077 2572 scope.go:117] "RemoveContainer" containerID="61c0d84400b9356994eedc8257ca01378a49b1fc7384ad87004a46e61ab14f02" Sep 10 04:43:46.097388 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.097364 2572 scope.go:117] "RemoveContainer" containerID="a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80" Sep 10 04:43:46.097809 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:46.097789 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80\": container with ID starting with a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80 not found: ID does not exist" containerID="a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80" Sep 10 04:43:46.097903 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.097823 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80"} err="failed to get container status \"a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80\": rpc error: code = NotFound desc = could not find container \"a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80\": container with ID starting with a6347088a5804145b9ccc36ab8c49d15d4a93e60c22f0119ccf25673fc9a4c80 not found: ID does not exist" Sep 10 04:43:46.097903 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.097850 2572 scope.go:117] "RemoveContainer" containerID="e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:46.098165 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb\": container with ID starting with e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb not found: ID does not exist" containerID="e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.098612 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb"} err="failed to get container status \"e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb\": rpc error: code = NotFound desc = could not find container \"e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb\": container with ID starting with e760a3229edd625b9b2bd66d8fad46f84741b4a4c2f9eb53a2bea229467005fb not found: ID does not exist" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.098704 2572 scope.go:117] "RemoveContainer" containerID="bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.099286 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100006 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="prometheus" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100033 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="prometheus" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100065 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="config-reloader" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100074 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="config-reloader" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100097 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="thanos-sidecar" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100105 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="thanos-sidecar" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100119 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy-web" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100128 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy-web" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100148 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="init-config-reloader" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100157 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="init-config-reloader" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100182 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100191 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100225 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy-thanos" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100234 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy-thanos" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100374 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="thanos-sidecar" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100392 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="config-reloader" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100402 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100409 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy-thanos" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100424 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="prometheus" Sep 10 04:43:46.101041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.100432 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" containerName="kube-rbac-proxy-web" Sep 10 04:43:46.104734 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:46.104697 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d\": container with ID starting with bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d not found: ID does not exist" containerID="bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d" Sep 10 04:43:46.104841 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.104748 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d"} err="failed to get container status \"bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d\": rpc error: code = NotFound desc = could not find container \"bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d\": container with ID starting with bc5cf3256fd33f80079a10cde6302c6389a2558a5bf50f9098d23b5a3b74957d not found: ID does not exist" Sep 10 04:43:46.104841 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.104773 2572 scope.go:117] "RemoveContainer" containerID="5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e" Sep 10 04:43:46.105163 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:46.105135 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e\": container with ID starting with 5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e not found: ID does not exist" containerID="5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e" Sep 10 04:43:46.105253 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.105167 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e"} err="failed to get container status \"5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e\": rpc error: code = NotFound desc = could not find container \"5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e\": container with ID starting with 5ea29e17f32e7986405f6e029356eb08101129c173f2fdb8b7d5e73181881a5e not found: ID does not exist" Sep 10 04:43:46.105253 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.105185 2572 scope.go:117] "RemoveContainer" containerID="a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146" Sep 10 04:43:46.105499 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:46.105481 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146\": container with ID starting with a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146 not found: ID does not exist" containerID="a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146" Sep 10 04:43:46.105556 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.105507 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146"} err="failed to get container status \"a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146\": rpc error: code = NotFound desc = could not find container \"a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146\": container with ID starting with a55b32febd9c9e54ff640cce0e1d0cbd961fd510d3673ec41c0ae9be326f0146 not found: ID does not exist" Sep 10 04:43:46.105556 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.105524 2572 scope.go:117] "RemoveContainer" containerID="eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d" Sep 10 04:43:46.105796 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:46.105779 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d\": container with ID starting with eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d not found: ID does not exist" containerID="eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d" Sep 10 04:43:46.105837 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.105800 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d"} err="failed to get container status \"eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d\": rpc error: code = NotFound desc = could not find container \"eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d\": container with ID starting with eacce4d49589f30be6f70d00e600bdcf61edc3dd06e360ffdd5b35ee96618d6d not found: ID does not exist" Sep 10 04:43:46.105837 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.105814 2572 scope.go:117] "RemoveContainer" containerID="61c0d84400b9356994eedc8257ca01378a49b1fc7384ad87004a46e61ab14f02" Sep 10 04:43:46.106093 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:43:46.106073 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c0d84400b9356994eedc8257ca01378a49b1fc7384ad87004a46e61ab14f02\": container with ID starting with 61c0d84400b9356994eedc8257ca01378a49b1fc7384ad87004a46e61ab14f02 not found: ID does not exist" containerID="61c0d84400b9356994eedc8257ca01378a49b1fc7384ad87004a46e61ab14f02" Sep 10 04:43:46.106148 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.106099 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c0d84400b9356994eedc8257ca01378a49b1fc7384ad87004a46e61ab14f02"} err="failed to get container status \"61c0d84400b9356994eedc8257ca01378a49b1fc7384ad87004a46e61ab14f02\": rpc error: code = NotFound desc = could not find container \"61c0d84400b9356994eedc8257ca01378a49b1fc7384ad87004a46e61ab14f02\": container with ID starting with 61c0d84400b9356994eedc8257ca01378a49b1fc7384ad87004a46e61ab14f02 not found: ID does not exist" Sep 10 04:43:46.109278 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.109257 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.112342 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.112314 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4f2wj\"" Sep 10 04:43:46.112493 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.112338 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 04:43:46.112493 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.112351 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 04:43:46.112493 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.112314 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 04:43:46.112493 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.112368 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 04:43:46.119380 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.119353 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 04:43:46.119380 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.119354 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 04:43:46.119737 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.119394 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 04:43:46.119737 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.119359 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 04:43:46.119737 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.119409 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 04:43:46.119916 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.119762 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 04:43:46.119916 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.119835 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6nh906m7lfq4a\"" Sep 10 04:43:46.119916 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.119844 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 04:43:46.119916 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.119890 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 04:43:46.125933 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.125904 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 04:43:46.140290 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.140261 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 04:43:46.146744 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.146715 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 04:43:46.205990 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.205953 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.205990 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.205997 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206206 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206024 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206206 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206091 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206206 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206110 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206206 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206128 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206212 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206238 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206259 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/517297a6-ebd6-4d0b-976e-707327e28eaa-config-out\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206277 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206293 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206307 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206343 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206379 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-web-config\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206416 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/517297a6-ebd6-4d0b-976e-707327e28eaa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206432 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206494 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-config\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206519 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206540 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ncdb\" (UniqueName: \"kubernetes.io/projected/517297a6-ebd6-4d0b-976e-707327e28eaa-kube-api-access-7ncdb\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.206772 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.206561 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.307820 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.307731 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.307820 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.307765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.307820 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.307783 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.307820 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.307801 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308124 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.307853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-web-config\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308124 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.307898 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/517297a6-ebd6-4d0b-976e-707327e28eaa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308124 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.307928 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308124 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.307963 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-config\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308124 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.307996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308124 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308025 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ncdb\" (UniqueName: \"kubernetes.io/projected/517297a6-ebd6-4d0b-976e-707327e28eaa-kube-api-access-7ncdb\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308124 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308057 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308124 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308093 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308575 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308575 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308155 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308575 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308191 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308575 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308220 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308575 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308244 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308575 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308575 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308305 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308575 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308334 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/517297a6-ebd6-4d0b-976e-707327e28eaa-config-out\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.308575 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308518 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.309023 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.308606 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.309100 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.309071 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.309295 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.309273 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.310910 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.310881 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.310910 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.310902 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/517297a6-ebd6-4d0b-976e-707327e28eaa-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.311338 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.311140 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-web-config\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.311766 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.311742 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.311879 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.311858 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.311952 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.311862 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.312016 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.311950 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/517297a6-ebd6-4d0b-976e-707327e28eaa-config-out\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.312358 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.312285 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/517297a6-ebd6-4d0b-976e-707327e28eaa-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.312441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.312357 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/517297a6-ebd6-4d0b-976e-707327e28eaa-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.312951 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.312926 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.313305 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.313279 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.313475 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.313427 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-config\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.313773 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.313754 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.314487 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.314465 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.314564 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.314482 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/517297a6-ebd6-4d0b-976e-707327e28eaa-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.319571 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.319550 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ncdb\" (UniqueName: \"kubernetes.io/projected/517297a6-ebd6-4d0b-976e-707327e28eaa-kube-api-access-7ncdb\") pod \"prometheus-k8s-0\" (UID: \"517297a6-ebd6-4d0b-976e-707327e28eaa\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.418887 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.418845 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:43:46.701921 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:46.701836 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 04:43:46.706108 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:43:46.706076 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517297a6_ebd6_4d0b_976e_707327e28eaa.slice/crio-15f91798e9fdae1253be496fcbc0aa8a9b584ba58fccfc14d715d6c996973941 WatchSource:0}: Error finding container 15f91798e9fdae1253be496fcbc0aa8a9b584ba58fccfc14d715d6c996973941: Status 404 returned error can't find the container with id 15f91798e9fdae1253be496fcbc0aa8a9b584ba58fccfc14d715d6c996973941 Sep 10 04:43:47.037258 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:47.037225 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41bcb40e-ba1d-4522-8428-4e811b2717fe" path="/var/lib/kubelet/pods/41bcb40e-ba1d-4522-8428-4e811b2717fe/volumes" Sep 10 04:43:47.041696 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:47.041663 2572 generic.go:356] "Generic (PLEG): container finished" podID="517297a6-ebd6-4d0b-976e-707327e28eaa" containerID="07abdf5d416594ef4a5d3bfdd3b61a0806c2241dc54c92b64895455ef88275b4" exitCode=0 Sep 10 04:43:47.041847 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:47.041744 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"517297a6-ebd6-4d0b-976e-707327e28eaa","Type":"ContainerDied","Data":"07abdf5d416594ef4a5d3bfdd3b61a0806c2241dc54c92b64895455ef88275b4"} Sep 10 04:43:47.041847 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:47.041821 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"517297a6-ebd6-4d0b-976e-707327e28eaa","Type":"ContainerStarted","Data":"15f91798e9fdae1253be496fcbc0aa8a9b584ba58fccfc14d715d6c996973941"} Sep 10 04:43:48.051281 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:48.051239 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"517297a6-ebd6-4d0b-976e-707327e28eaa","Type":"ContainerStarted","Data":"06111b73e0903958b9d4a1c94f7a7fb66315dfd2877eeaa0bc6005de213ece51"} Sep 10 04:43:48.051681 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:48.051312 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"517297a6-ebd6-4d0b-976e-707327e28eaa","Type":"ContainerStarted","Data":"42f56a27b7643cf6e6dac17f078e34161021e06f291099fec09a8d37a19da810"} Sep 10 04:43:48.051681 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:48.051353 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"517297a6-ebd6-4d0b-976e-707327e28eaa","Type":"ContainerStarted","Data":"4636c2509393a5bf0feabe594f8b9bffc80f4a9fb165ae024774510a85dce42b"} Sep 10 04:43:48.051681 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:48.051382 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"517297a6-ebd6-4d0b-976e-707327e28eaa","Type":"ContainerStarted","Data":"8922792c8c0d3a53bd8919f1ba1de281f0a6037165a01cb036964c13fe054912"} Sep 10 04:43:48.051681 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:48.051404 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"517297a6-ebd6-4d0b-976e-707327e28eaa","Type":"ContainerStarted","Data":"e787d2ba23d7e607618746a6a93a4f50cebfa89144a7597be47a07825c2f1a16"} Sep 10 04:43:48.051681 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:48.051439 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"517297a6-ebd6-4d0b-976e-707327e28eaa","Type":"ContainerStarted","Data":"5df62ca4db87a8686bb49eb021c93f6f444b82b781ab225405a32e9fd3165a67"} Sep 10 04:43:48.083036 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:48.082961 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.063597441 podStartE2EDuration="2.082932649s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.335208ms" isStatefulPod=false podCreationTimestamp="2026-09-10 04:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:43:48.081665452 +0000 UTC m=+147.585498541" watchObservedRunningTime="2026-09-10 04:43:48.082932649 +0000 UTC m=+147.586765713" Sep 10 04:43:51.419918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:43:51.419858 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:44:46.419404 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:44:46.419357 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:44:46.435602 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:44:46.435573 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:44:47.316661 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:44:47.316636 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 04:46:20.942177 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:20.942146 2572 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 04:46:50.466855 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.466816 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-52c94"] Sep 10 04:46:50.470069 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.470051 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" Sep 10 04:46:50.472719 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.472682 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 04:46:50.473676 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.473654 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 04:46:50.473745 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.473659 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jbhx5\"" Sep 10 04:46:50.480334 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.480304 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-52c94"] Sep 10 04:46:50.595842 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.595799 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcrjh\" (UniqueName: \"kubernetes.io/projected/2efa999b-bcb3-4563-939a-a55ecaf373c9-kube-api-access-wcrjh\") pod \"cert-manager-webhook-6486ffd997-52c94\" (UID: \"2efa999b-bcb3-4563-939a-a55ecaf373c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" Sep 10 04:46:50.595842 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.595843 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2efa999b-bcb3-4563-939a-a55ecaf373c9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-52c94\" (UID: \"2efa999b-bcb3-4563-939a-a55ecaf373c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" Sep 10 04:46:50.696640 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.696593 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcrjh\" (UniqueName: \"kubernetes.io/projected/2efa999b-bcb3-4563-939a-a55ecaf373c9-kube-api-access-wcrjh\") pod \"cert-manager-webhook-6486ffd997-52c94\" (UID: \"2efa999b-bcb3-4563-939a-a55ecaf373c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" Sep 10 04:46:50.696640 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.696644 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2efa999b-bcb3-4563-939a-a55ecaf373c9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-52c94\" (UID: \"2efa999b-bcb3-4563-939a-a55ecaf373c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" Sep 10 04:46:50.705062 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.705022 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2efa999b-bcb3-4563-939a-a55ecaf373c9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-52c94\" (UID: \"2efa999b-bcb3-4563-939a-a55ecaf373c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" Sep 10 04:46:50.705206 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.705108 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcrjh\" (UniqueName: \"kubernetes.io/projected/2efa999b-bcb3-4563-939a-a55ecaf373c9-kube-api-access-wcrjh\") pod \"cert-manager-webhook-6486ffd997-52c94\" (UID: \"2efa999b-bcb3-4563-939a-a55ecaf373c9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" Sep 10 04:46:50.793335 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:50.793304 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" Sep 10 04:46:51.066912 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:51.066832 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-52c94"] Sep 10 04:46:51.070071 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:46:51.070043 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2efa999b_bcb3_4563_939a_a55ecaf373c9.slice/crio-a9ce9fcc48f7b4c7812f5594c708237b82a958b19de5c031b058aa279e87eae9 WatchSource:0}: Error finding container a9ce9fcc48f7b4c7812f5594c708237b82a958b19de5c031b058aa279e87eae9: Status 404 returned error can't find the container with id a9ce9fcc48f7b4c7812f5594c708237b82a958b19de5c031b058aa279e87eae9 Sep 10 04:46:51.072040 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:51.072022 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 04:46:51.799172 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:51.799134 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" event={"ID":"2efa999b-bcb3-4563-939a-a55ecaf373c9","Type":"ContainerStarted","Data":"a9ce9fcc48f7b4c7812f5594c708237b82a958b19de5c031b058aa279e87eae9"} Sep 10 04:46:54.814826 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:54.814691 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" event={"ID":"2efa999b-bcb3-4563-939a-a55ecaf373c9","Type":"ContainerStarted","Data":"f84264bed2b854fa033705344eb4748e48bac33138f4986ce51df3c9a6bd2e2d"} Sep 10 04:46:54.815288 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:54.814874 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" Sep 10 04:46:54.840826 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:46:54.840763 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" podStartSLOduration=1.188170641 podStartE2EDuration="4.840744012s" totalImagesPullingTime="3.652573371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:46:54.839389369 +0000 UTC m=+334.343222429" watchObservedRunningTime="2026-09-10 04:46:54.840744012 +0000 UTC m=+334.344577073" Sep 10 04:47:00.821924 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:00.821883 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-52c94" Sep 10 04:47:03.968637 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:03.968600 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q"] Sep 10 04:47:03.972047 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:03.972024 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q" Sep 10 04:47:03.974385 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:03.974359 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 04:47:03.975506 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:03.975487 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9k46b\"" Sep 10 04:47:03.975605 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:03.975541 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 04:47:03.980349 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:03.980317 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q"] Sep 10 04:47:04.114339 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:04.114299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b3454d-b1dd-4681-8b47-6e7222779d75-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tn44q\" (UID: \"48b3454d-b1dd-4681-8b47-6e7222779d75\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q" Sep 10 04:47:04.114550 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:04.114356 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27zrk\" (UniqueName: \"kubernetes.io/projected/48b3454d-b1dd-4681-8b47-6e7222779d75-kube-api-access-27zrk\") pod \"openshift-lws-operator-fd8ccff4c-tn44q\" (UID: \"48b3454d-b1dd-4681-8b47-6e7222779d75\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q" Sep 10 04:47:04.215550 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:04.215509 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27zrk\" (UniqueName: \"kubernetes.io/projected/48b3454d-b1dd-4681-8b47-6e7222779d75-kube-api-access-27zrk\") pod \"openshift-lws-operator-fd8ccff4c-tn44q\" (UID: \"48b3454d-b1dd-4681-8b47-6e7222779d75\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q" Sep 10 04:47:04.215724 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:04.215592 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b3454d-b1dd-4681-8b47-6e7222779d75-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tn44q\" (UID: \"48b3454d-b1dd-4681-8b47-6e7222779d75\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q" Sep 10 04:47:04.215964 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:04.215944 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b3454d-b1dd-4681-8b47-6e7222779d75-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tn44q\" (UID: \"48b3454d-b1dd-4681-8b47-6e7222779d75\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q" Sep 10 04:47:04.223119 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:04.223062 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27zrk\" (UniqueName: \"kubernetes.io/projected/48b3454d-b1dd-4681-8b47-6e7222779d75-kube-api-access-27zrk\") pod \"openshift-lws-operator-fd8ccff4c-tn44q\" (UID: \"48b3454d-b1dd-4681-8b47-6e7222779d75\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q" Sep 10 04:47:04.281528 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:04.281490 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q" Sep 10 04:47:04.555245 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:04.555199 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q"] Sep 10 04:47:04.557502 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:47:04.557470 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48b3454d_b1dd_4681_8b47_6e7222779d75.slice/crio-8eb29de7feb7fe50ffa273c92a4d6151526c1f37e35eafeb141e4c37f4b16dfd WatchSource:0}: Error finding container 8eb29de7feb7fe50ffa273c92a4d6151526c1f37e35eafeb141e4c37f4b16dfd: Status 404 returned error can't find the container with id 8eb29de7feb7fe50ffa273c92a4d6151526c1f37e35eafeb141e4c37f4b16dfd Sep 10 04:47:04.858404 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:04.858316 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q" event={"ID":"48b3454d-b1dd-4681-8b47-6e7222779d75","Type":"ContainerStarted","Data":"8eb29de7feb7fe50ffa273c92a4d6151526c1f37e35eafeb141e4c37f4b16dfd"} Sep 10 04:47:07.872563 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:07.872522 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q" event={"ID":"48b3454d-b1dd-4681-8b47-6e7222779d75","Type":"ContainerStarted","Data":"0a77e3612500f802f9d1b428d61b097a1b6ed5cb76215eb007094996cf59f82d"} Sep 10 04:47:07.890600 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:07.890542 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tn44q" podStartSLOduration=2.579026862 podStartE2EDuration="4.890523319s" totalImagesPullingTime="2.311496457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:47:07.888546885 +0000 UTC m=+347.392379947" watchObservedRunningTime="2026-09-10 04:47:07.890523319 +0000 UTC m=+347.394356380" Sep 10 04:47:27.430361 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.430271 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8"] Sep 10 04:47:27.479674 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.479621 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8"] Sep 10 04:47:27.479848 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.479646 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.482551 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.482522 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 04:47:27.482551 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.482551 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dzbdb\"" Sep 10 04:47:27.482778 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.482522 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 04:47:27.482778 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.482581 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 04:47:27.482778 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.482630 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 04:47:27.525831 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.525787 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/75eeed64-56c0-47e8-a25f-c61c067b69e3-manifests\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.525831 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.525829 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75eeed64-56c0-47e8-a25f-c61c067b69e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.526073 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.525865 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/75eeed64-56c0-47e8-a25f-c61c067b69e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.526073 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.525929 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn22j\" (UniqueName: \"kubernetes.io/projected/75eeed64-56c0-47e8-a25f-c61c067b69e3-kube-api-access-qn22j\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.526073 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.525948 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75eeed64-56c0-47e8-a25f-c61c067b69e3-tmp\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.627109 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.627073 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/75eeed64-56c0-47e8-a25f-c61c067b69e3-manifests\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.627109 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.627111 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75eeed64-56c0-47e8-a25f-c61c067b69e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.627330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.627145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/75eeed64-56c0-47e8-a25f-c61c067b69e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.627330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.627191 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn22j\" (UniqueName: \"kubernetes.io/projected/75eeed64-56c0-47e8-a25f-c61c067b69e3-kube-api-access-qn22j\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.627330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.627212 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75eeed64-56c0-47e8-a25f-c61c067b69e3-tmp\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.627568 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.627544 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/75eeed64-56c0-47e8-a25f-c61c067b69e3-manifests\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.629524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.629496 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/75eeed64-56c0-47e8-a25f-c61c067b69e3-tmp\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.629730 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.629712 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/75eeed64-56c0-47e8-a25f-c61c067b69e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.629775 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.629725 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/75eeed64-56c0-47e8-a25f-c61c067b69e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.634965 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.634936 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn22j\" (UniqueName: \"kubernetes.io/projected/75eeed64-56c0-47e8-a25f-c61c067b69e3-kube-api-access-qn22j\") pod \"opendatahub-operator-controller-manager-c57f4866c-g8gh8\" (UID: \"75eeed64-56c0-47e8-a25f-c61c067b69e3\") " pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:27.790872 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:27.790832 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:28.088747 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:28.088665 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8"] Sep 10 04:47:28.092481 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:47:28.092427 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75eeed64_56c0_47e8_a25f_c61c067b69e3.slice/crio-91b388fce9c3e1d8b95346014c756be1d81056aa5801845d4cb9179003886d46 WatchSource:0}: Error finding container 91b388fce9c3e1d8b95346014c756be1d81056aa5801845d4cb9179003886d46: Status 404 returned error can't find the container with id 91b388fce9c3e1d8b95346014c756be1d81056aa5801845d4cb9179003886d46 Sep 10 04:47:28.965701 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:28.965655 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" event={"ID":"75eeed64-56c0-47e8-a25f-c61c067b69e3","Type":"ContainerStarted","Data":"91b388fce9c3e1d8b95346014c756be1d81056aa5801845d4cb9179003886d46"} Sep 10 04:47:31.982124 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:31.982087 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" event={"ID":"75eeed64-56c0-47e8-a25f-c61c067b69e3","Type":"ContainerStarted","Data":"c5735bf30817a3b7555a307d34d001e39c13a9ba6c384dbf0a953d7902ad4808"} Sep 10 04:47:32.988399 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:32.988365 2572 generic.go:356] "Generic (PLEG): container finished" podID="75eeed64-56c0-47e8-a25f-c61c067b69e3" containerID="c5735bf30817a3b7555a307d34d001e39c13a9ba6c384dbf0a953d7902ad4808" exitCode=0 Sep 10 04:47:32.988810 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:32.988441 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" event={"ID":"75eeed64-56c0-47e8-a25f-c61c067b69e3","Type":"ContainerDied","Data":"c5735bf30817a3b7555a307d34d001e39c13a9ba6c384dbf0a953d7902ad4808"} Sep 10 04:47:33.995306 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:33.995264 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" event={"ID":"75eeed64-56c0-47e8-a25f-c61c067b69e3","Type":"ContainerStarted","Data":"98f53eb04cdce8028d05b80a494b2479e4ac3a3d1cc20ed7ba549e9976db81c1"} Sep 10 04:47:33.995732 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:33.995415 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:34.017187 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:34.017132 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" podStartSLOduration=2.471197725 podStartE2EDuration="7.017115719s" totalImagesPullingTime="3.427247457s" totalInitContainerRuntime="1.118670537s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:47:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:47:34.015566385 +0000 UTC m=+373.519399446" watchObservedRunningTime="2026-09-10 04:47:34.017115719 +0000 UTC m=+373.520948779" Sep 10 04:47:45.002537 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:45.002498 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c57f4866c-g8gh8" Sep 10 04:47:54.058919 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.058883 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp"] Sep 10 04:47:54.062402 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.062381 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.064888 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.064863 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 04:47:54.065035 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.064860 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-t6jgh\"" Sep 10 04:47:54.065035 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.065006 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 04:47:54.065218 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.065203 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 04:47:54.072644 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.072616 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp"] Sep 10 04:47:54.076890 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.076855 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxpr7\" (UniqueName: \"kubernetes.io/projected/c37350ae-7be6-421a-a2ce-0ee47bacd479-kube-api-access-lxpr7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.077035 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.076904 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.077035 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.076960 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.077035 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.076994 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.077035 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.077013 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.077035 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.077033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37350ae-7be6-421a-a2ce-0ee47bacd479-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.077209 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.077103 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.077209 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.077131 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.077209 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.077157 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.177725 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.177683 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.177937 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.177737 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.177937 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.177757 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.177937 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.177780 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37350ae-7be6-421a-a2ce-0ee47bacd479-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.177937 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.177887 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.177937 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.177928 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.178211 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.177961 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.178211 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.178003 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpr7\" (UniqueName: \"kubernetes.io/projected/c37350ae-7be6-421a-a2ce-0ee47bacd479-kube-api-access-lxpr7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.178211 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.178054 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.178411 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.178332 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.178411 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.178358 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.178411 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.178399 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.178593 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.178487 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.178593 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.178585 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c37350ae-7be6-421a-a2ce-0ee47bacd479-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.180145 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.180123 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.180380 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.180362 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.186271 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.186239 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c37350ae-7be6-421a-a2ce-0ee47bacd479-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.186433 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.186358 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxpr7\" (UniqueName: \"kubernetes.io/projected/c37350ae-7be6-421a-a2ce-0ee47bacd479-kube-api-access-lxpr7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp\" (UID: \"c37350ae-7be6-421a-a2ce-0ee47bacd479\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.373865 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.373762 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:54.651705 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:54.651678 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp"] Sep 10 04:47:54.653740 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:47:54.653715 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37350ae_7be6_421a_a2ce_0ee47bacd479.slice/crio-e85c0a9dd0b37d7752b6ea0d7d6436a557688995b87d10596f59f4acbad32ffe WatchSource:0}: Error finding container e85c0a9dd0b37d7752b6ea0d7d6436a557688995b87d10596f59f4acbad32ffe: Status 404 returned error can't find the container with id e85c0a9dd0b37d7752b6ea0d7d6436a557688995b87d10596f59f4acbad32ffe Sep 10 04:47:55.092573 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:55.092540 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" event={"ID":"c37350ae-7be6-421a-a2ce-0ee47bacd479","Type":"ContainerStarted","Data":"e85c0a9dd0b37d7752b6ea0d7d6436a557688995b87d10596f59f4acbad32ffe"} Sep 10 04:47:57.312600 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:57.312558 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:47:57.312993 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:57.312643 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:47:57.312993 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:57.312696 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:47:58.110046 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:58.110009 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" event={"ID":"c37350ae-7be6-421a-a2ce-0ee47bacd479","Type":"ContainerStarted","Data":"423bb5340332b45016bcede5243e7fb3af1ad0b1e38d7ea8c4a90b3622508aae"} Sep 10 04:47:58.134437 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:58.134380 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" podStartSLOduration=1.47793996 podStartE2EDuration="4.134364106s" totalImagesPullingTime="2.656424146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:47:58.133576464 +0000 UTC m=+397.637409527" watchObservedRunningTime="2026-09-10 04:47:58.134364106 +0000 UTC m=+397.638197167" Sep 10 04:47:58.374371 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:58.374281 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:58.379394 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:58.379367 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:59.114596 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:59.114567 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:47:59.115752 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:47:59.115732 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp" Sep 10 04:48:05.283691 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.283652 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt"] Sep 10 04:48:05.291102 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.291072 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.293916 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.293882 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 04:48:05.294070 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.293889 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 04:48:05.295017 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.294998 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kghkv\"" Sep 10 04:48:05.295698 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.295674 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 04:48:05.304464 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.304412 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt"] Sep 10 04:48:05.386228 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.386179 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvk94\" (UniqueName: \"kubernetes.io/projected/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-kube-api-access-bvk94\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.386410 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.386258 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.386410 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.386327 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.386410 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.386348 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.386410 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.386380 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.386657 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.386424 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.386657 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.386567 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.487812 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.487768 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.488024 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.487856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvk94\" (UniqueName: \"kubernetes.io/projected/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-kube-api-access-bvk94\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.488024 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.487897 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.488024 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.487940 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.488024 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.487967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.488024 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.488005 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.488297 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.488120 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.489083 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.489049 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.490881 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.490845 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.491006 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.490877 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.491074 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.491053 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.491130 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.491089 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.496244 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.496211 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.496528 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.496507 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvk94\" (UniqueName: \"kubernetes.io/projected/fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b-kube-api-access-bvk94\") pod \"istiod-openshift-gateway-5677ccb49b-dgxnt\" (UID: \"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.603111 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.603028 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:05.662742 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.662663 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd"] Sep 10 04:48:05.672095 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.671764 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.675676 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.675144 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 04:48:05.675676 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.675204 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 04:48:05.677639 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.675551 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 04:48:05.677639 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.677486 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd"] Sep 10 04:48:05.677639 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.675607 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hv8zz\"" Sep 10 04:48:05.791177 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.791137 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/469671b2-66b7-4971-9bcc-6c79151f8072-cert\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.791349 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.791190 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/469671b2-66b7-4971-9bcc-6c79151f8072-manager-config\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.791349 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.791206 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/469671b2-66b7-4971-9bcc-6c79151f8072-metrics-cert\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.791349 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.791295 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chpj\" (UniqueName: \"kubernetes.io/projected/469671b2-66b7-4971-9bcc-6c79151f8072-kube-api-access-2chpj\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.892635 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.892542 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chpj\" (UniqueName: \"kubernetes.io/projected/469671b2-66b7-4971-9bcc-6c79151f8072-kube-api-access-2chpj\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.892635 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.892623 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/469671b2-66b7-4971-9bcc-6c79151f8072-cert\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.892887 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.892659 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/469671b2-66b7-4971-9bcc-6c79151f8072-manager-config\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.892887 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.892676 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/469671b2-66b7-4971-9bcc-6c79151f8072-metrics-cert\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.893417 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.893391 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/469671b2-66b7-4971-9bcc-6c79151f8072-manager-config\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.895763 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.895739 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/469671b2-66b7-4971-9bcc-6c79151f8072-cert\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.895763 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.895759 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/469671b2-66b7-4971-9bcc-6c79151f8072-metrics-cert\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.904260 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.904229 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt"] Sep 10 04:48:05.908011 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:48:05.907983 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae80c42_bbf1_4d41_a1d4_87d5dfd4ee1b.slice/crio-dbfac9a1e1951a1b4a0d5b3baab2845762caa572a7b34d06c3c95609567c266e WatchSource:0}: Error finding container dbfac9a1e1951a1b4a0d5b3baab2845762caa572a7b34d06c3c95609567c266e: Status 404 returned error can't find the container with id dbfac9a1e1951a1b4a0d5b3baab2845762caa572a7b34d06c3c95609567c266e Sep 10 04:48:05.915474 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.915417 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chpj\" (UniqueName: \"kubernetes.io/projected/469671b2-66b7-4971-9bcc-6c79151f8072-kube-api-access-2chpj\") pod \"lws-controller-manager-5fb77b59f6-6lnfd\" (UID: \"469671b2-66b7-4971-9bcc-6c79151f8072\") " pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:05.991720 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:05.991677 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:06.148736 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:06.148642 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" event={"ID":"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b","Type":"ContainerStarted","Data":"dbfac9a1e1951a1b4a0d5b3baab2845762caa572a7b34d06c3c95609567c266e"} Sep 10 04:48:06.274006 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:06.273978 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd"] Sep 10 04:48:06.276565 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:48:06.276538 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469671b2_66b7_4971_9bcc_6c79151f8072.slice/crio-649f80fb4567bd7a9cd1c9cb5a1e00c6383b8c73f282a0fb088fbef5982109f1 WatchSource:0}: Error finding container 649f80fb4567bd7a9cd1c9cb5a1e00c6383b8c73f282a0fb088fbef5982109f1: Status 404 returned error can't find the container with id 649f80fb4567bd7a9cd1c9cb5a1e00c6383b8c73f282a0fb088fbef5982109f1 Sep 10 04:48:07.157569 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:07.157492 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" event={"ID":"469671b2-66b7-4971-9bcc-6c79151f8072","Type":"ContainerStarted","Data":"649f80fb4567bd7a9cd1c9cb5a1e00c6383b8c73f282a0fb088fbef5982109f1"} Sep 10 04:48:08.656902 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:08.656840 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:48:08.657233 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:08.656907 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:48:09.171051 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:09.170989 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" event={"ID":"fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b","Type":"ContainerStarted","Data":"720f21200679d0139f26a413a81f598a83de4b1ff6463b19f0f61cac6916d92f"} Sep 10 04:48:09.171321 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:09.171233 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:09.173227 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:09.173177 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" event={"ID":"469671b2-66b7-4971-9bcc-6c79151f8072","Type":"ContainerStarted","Data":"af966d38c2bab9bfdc85ff2574de69f337cbef2433910be35c6ef5ec0aa3717b"} Sep 10 04:48:09.173428 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:09.173267 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:48:09.173428 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:09.173278 2572 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-dgxnt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 04:48:09.173428 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:09.173337 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" podUID="fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 04:48:09.227057 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:09.227001 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" podStartSLOduration=1.480783677 podStartE2EDuration="4.226982159s" totalImagesPullingTime="2.746198482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:48:09.207657087 +0000 UTC m=+408.711490150" watchObservedRunningTime="2026-09-10 04:48:09.226982159 +0000 UTC m=+408.730815220" Sep 10 04:48:09.227329 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:09.227309 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" podStartSLOduration=1.8472956059999999 podStartE2EDuration="4.227304178s" totalImagesPullingTime="2.380008572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:48:09.225786145 +0000 UTC m=+408.729619220" watchObservedRunningTime="2026-09-10 04:48:09.227304178 +0000 UTC m=+408.731137314" Sep 10 04:48:10.179374 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:10.179340 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dgxnt" Sep 10 04:48:20.180594 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:48:20.180558 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5fb77b59f6-6lnfd" Sep 10 04:50:09.721113 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.721078 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw"] Sep 10 04:50:09.724609 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.724586 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.727306 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.727274 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-khd29\"" Sep 10 04:50:09.735579 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.735541 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw"] Sep 10 04:50:09.758656 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.758619 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716fa502-5c07-487c-9410-4546ddfffec2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.758842 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.758669 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbl5z\" (UniqueName: \"kubernetes.io/projected/716fa502-5c07-487c-9410-4546ddfffec2-kube-api-access-qbl5z\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.758842 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.758748 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/716fa502-5c07-487c-9410-4546ddfffec2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.758842 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.758771 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.758842 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.758815 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.758842 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.758830 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.759070 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.758847 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.759070 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.758871 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.759070 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.758889 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716fa502-5c07-487c-9410-4546ddfffec2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.859571 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.859523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.859777 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.859599 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.859777 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.859622 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.859777 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.859643 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.859777 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.859679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.859777 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.859702 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716fa502-5c07-487c-9410-4546ddfffec2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.859777 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.859759 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716fa502-5c07-487c-9410-4546ddfffec2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.860075 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.859799 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbl5z\" (UniqueName: \"kubernetes.io/projected/716fa502-5c07-487c-9410-4546ddfffec2-kube-api-access-qbl5z\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.860075 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.859868 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/716fa502-5c07-487c-9410-4546ddfffec2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.860174 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.860077 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.860174 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.860111 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.860174 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.860138 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.860327 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.860185 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.860789 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.860767 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716fa502-5c07-487c-9410-4546ddfffec2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.862062 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.862036 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716fa502-5c07-487c-9410-4546ddfffec2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.862368 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.862346 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716fa502-5c07-487c-9410-4546ddfffec2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.868682 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.868653 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/716fa502-5c07-487c-9410-4546ddfffec2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:09.868909 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:09.868883 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbl5z\" (UniqueName: \"kubernetes.io/projected/716fa502-5c07-487c-9410-4546ddfffec2-kube-api-access-qbl5z\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9l5xw\" (UID: \"716fa502-5c07-487c-9410-4546ddfffec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:10.037129 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:10.037084 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:10.316057 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:10.316027 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw"] Sep 10 04:50:10.318327 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:50:10.318297 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716fa502_5c07_487c_9410_4546ddfffec2.slice/crio-190355b885ffa6f2aeb0c4e3676417c95dc945f781ace115f201b38fbf9843cc WatchSource:0}: Error finding container 190355b885ffa6f2aeb0c4e3676417c95dc945f781ace115f201b38fbf9843cc: Status 404 returned error can't find the container with id 190355b885ffa6f2aeb0c4e3676417c95dc945f781ace115f201b38fbf9843cc Sep 10 04:50:10.320511 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:10.320469 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:50:10.320622 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:10.320563 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:50:10.320622 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:10.320612 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:50:10.756286 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:10.756240 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" event={"ID":"716fa502-5c07-487c-9410-4546ddfffec2","Type":"ContainerStarted","Data":"24a8f96ef336ebbb7405d3d818f79138341906f95bbb97fdd1ac09f2d3cf2f5b"} Sep 10 04:50:10.756755 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:10.756308 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" event={"ID":"716fa502-5c07-487c-9410-4546ddfffec2","Type":"ContainerStarted","Data":"190355b885ffa6f2aeb0c4e3676417c95dc945f781ace115f201b38fbf9843cc"} Sep 10 04:50:10.779465 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:10.779385 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" podStartSLOduration=1.7793667119999999 podStartE2EDuration="1.779366712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:50:10.778272824 +0000 UTC m=+530.282105887" watchObservedRunningTime="2026-09-10 04:50:10.779366712 +0000 UTC m=+530.283199773" Sep 10 04:50:11.038143 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:11.038051 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:11.039779 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:11.039745 2572 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9l5xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 10 04:50:11.039927 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:11.039826 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" podUID="716fa502-5c07-487c-9410-4546ddfffec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 10 04:50:12.041707 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:12.041669 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:12.766689 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:12.766655 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:50:12.767686 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:50:12.767665 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:51:01.603818 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.603780 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b87c56bcc-d2mhs"] Sep 10 04:51:01.607408 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.607386 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:01.609787 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.609764 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 04:51:01.609787 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.609782 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 04:51:01.609965 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.609846 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 04:51:01.610989 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.610965 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z9tjc\"" Sep 10 04:51:01.617629 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.617598 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b87c56bcc-d2mhs"] Sep 10 04:51:01.745605 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.745571 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-webhook-cert\") pod \"maas-controller-5b87c56bcc-d2mhs\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:01.745769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.745647 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lss69\" (UniqueName: \"kubernetes.io/projected/c984b1ee-63fb-424c-93c2-add37aaa4cc2-kube-api-access-lss69\") pod \"maas-controller-5b87c56bcc-d2mhs\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:01.745769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.745692 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-metrics-certs\") pod \"maas-controller-5b87c56bcc-d2mhs\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:01.846401 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.846361 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lss69\" (UniqueName: \"kubernetes.io/projected/c984b1ee-63fb-424c-93c2-add37aaa4cc2-kube-api-access-lss69\") pod \"maas-controller-5b87c56bcc-d2mhs\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:01.846585 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.846424 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-metrics-certs\") pod \"maas-controller-5b87c56bcc-d2mhs\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:01.846585 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.846485 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-webhook-cert\") pod \"maas-controller-5b87c56bcc-d2mhs\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:01.846585 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:51:01.846575 2572 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 10 04:51:01.846699 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:51:01.846635 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-webhook-cert podName:c984b1ee-63fb-424c-93c2-add37aaa4cc2 nodeName:}" failed. No retries permitted until 2026-09-10 04:51:02.346619922 +0000 UTC m=+581.850452961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-webhook-cert") pod "maas-controller-5b87c56bcc-d2mhs" (UID: "c984b1ee-63fb-424c-93c2-add37aaa4cc2") : secret "maas-controller-webhook-cert" not found Sep 10 04:51:01.848870 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.848847 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-metrics-certs\") pod \"maas-controller-5b87c56bcc-d2mhs\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:01.854122 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:01.854053 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lss69\" (UniqueName: \"kubernetes.io/projected/c984b1ee-63fb-424c-93c2-add37aaa4cc2-kube-api-access-lss69\") pod \"maas-controller-5b87c56bcc-d2mhs\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:02.351264 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:02.351222 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-webhook-cert\") pod \"maas-controller-5b87c56bcc-d2mhs\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:02.353558 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:02.353533 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-webhook-cert\") pod \"maas-controller-5b87c56bcc-d2mhs\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:02.519273 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:02.519228 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:02.790320 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:02.790290 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b87c56bcc-d2mhs"] Sep 10 04:51:02.792878 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:51:02.792852 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc984b1ee_63fb_424c_93c2_add37aaa4cc2.slice/crio-2fe0ee1e3faba7ad86eae3a607b5aaa6e2fdc9459d3ccc022b26cb8717b87061 WatchSource:0}: Error finding container 2fe0ee1e3faba7ad86eae3a607b5aaa6e2fdc9459d3ccc022b26cb8717b87061: Status 404 returned error can't find the container with id 2fe0ee1e3faba7ad86eae3a607b5aaa6e2fdc9459d3ccc022b26cb8717b87061 Sep 10 04:51:03.011286 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:03.011252 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" event={"ID":"c984b1ee-63fb-424c-93c2-add37aaa4cc2","Type":"ContainerStarted","Data":"2fe0ee1e3faba7ad86eae3a607b5aaa6e2fdc9459d3ccc022b26cb8717b87061"} Sep 10 04:51:06.031772 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:06.031737 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" event={"ID":"c984b1ee-63fb-424c-93c2-add37aaa4cc2","Type":"ContainerStarted","Data":"9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c"} Sep 10 04:51:06.032190 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:06.031930 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:06.048755 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:06.048704 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" podStartSLOduration=2.501896478 podStartE2EDuration="5.048689509s" totalImagesPullingTime="2.546793031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:51:06.04801591 +0000 UTC m=+585.551848967" watchObservedRunningTime="2026-09-10 04:51:06.048689509 +0000 UTC m=+585.552522570" Sep 10 04:51:16.785843 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:16.785804 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b87c56bcc-d2mhs"] Sep 10 04:51:16.786279 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:16.786028 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" podUID="c984b1ee-63fb-424c-93c2-add37aaa4cc2" containerName="manager" containerID="cri-o://9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c" gracePeriod=30 Sep 10 04:51:16.787732 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:16.787712 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:17.036257 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.036194 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:17.089418 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.089376 2572 generic.go:356] "Generic (PLEG): container finished" podID="c984b1ee-63fb-424c-93c2-add37aaa4cc2" containerID="9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c" exitCode=0 Sep 10 04:51:17.089622 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.089491 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" event={"ID":"c984b1ee-63fb-424c-93c2-add37aaa4cc2","Type":"ContainerDied","Data":"9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c"} Sep 10 04:51:17.089622 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.089531 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" Sep 10 04:51:17.089622 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.089587 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b87c56bcc-d2mhs" event={"ID":"c984b1ee-63fb-424c-93c2-add37aaa4cc2","Type":"ContainerDied","Data":"2fe0ee1e3faba7ad86eae3a607b5aaa6e2fdc9459d3ccc022b26cb8717b87061"} Sep 10 04:51:17.089725 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.089624 2572 scope.go:117] "RemoveContainer" containerID="9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c" Sep 10 04:51:17.095540 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.095514 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lss69\" (UniqueName: \"kubernetes.io/projected/c984b1ee-63fb-424c-93c2-add37aaa4cc2-kube-api-access-lss69\") pod \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " Sep 10 04:51:17.095706 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.095594 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-metrics-certs\") pod \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " Sep 10 04:51:17.095706 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.095629 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-webhook-cert\") pod \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\" (UID: \"c984b1ee-63fb-424c-93c2-add37aaa4cc2\") " Sep 10 04:51:17.098283 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.098241 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c984b1ee-63fb-424c-93c2-add37aaa4cc2-kube-api-access-lss69" (OuterVolumeSpecName: "kube-api-access-lss69") pod "c984b1ee-63fb-424c-93c2-add37aaa4cc2" (UID: "c984b1ee-63fb-424c-93c2-add37aaa4cc2"). InnerVolumeSpecName "kube-api-access-lss69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:51:17.098397 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.098242 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c984b1ee-63fb-424c-93c2-add37aaa4cc2" (UID: "c984b1ee-63fb-424c-93c2-add37aaa4cc2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:51:17.098397 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.098321 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c984b1ee-63fb-424c-93c2-add37aaa4cc2" (UID: "c984b1ee-63fb-424c-93c2-add37aaa4cc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:51:17.099387 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.099367 2572 scope.go:117] "RemoveContainer" containerID="9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c" Sep 10 04:51:17.099790 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:51:17.099768 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c\": container with ID starting with 9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c not found: ID does not exist" containerID="9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c" Sep 10 04:51:17.099875 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.099799 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c"} err="failed to get container status \"9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c\": rpc error: code = NotFound desc = could not find container \"9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c\": container with ID starting with 9287cecf2539b4b34304e598654c318f4f2af2354f1797fbcf824d9360fee75c not found: ID does not exist" Sep 10 04:51:17.196511 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.196440 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:51:17.196511 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.196505 2572 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c984b1ee-63fb-424c-93c2-add37aaa4cc2-webhook-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:51:17.196511 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.196515 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lss69\" (UniqueName: \"kubernetes.io/projected/c984b1ee-63fb-424c-93c2-add37aaa4cc2-kube-api-access-lss69\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:51:17.412720 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.412684 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b87c56bcc-d2mhs"] Sep 10 04:51:17.418281 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:17.418241 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b87c56bcc-d2mhs"] Sep 10 04:51:19.036705 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:19.036670 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c984b1ee-63fb-424c-93c2-add37aaa4cc2" path="/var/lib/kubelet/pods/c984b1ee-63fb-424c-93c2-add37aaa4cc2/volumes" Sep 10 04:51:38.912348 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.912307 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5"] Sep 10 04:51:38.912912 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.912727 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c984b1ee-63fb-424c-93c2-add37aaa4cc2" containerName="manager" Sep 10 04:51:38.912912 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.912741 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c984b1ee-63fb-424c-93c2-add37aaa4cc2" containerName="manager" Sep 10 04:51:38.912912 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.912831 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c984b1ee-63fb-424c-93c2-add37aaa4cc2" containerName="manager" Sep 10 04:51:38.920881 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.920848 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:38.923840 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.923809 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 04:51:38.924023 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.923902 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 04:51:38.924108 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.924069 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 04:51:38.924208 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.924187 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 04:51:38.924881 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.924859 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksvhk\"" Sep 10 04:51:38.925579 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.925557 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5"] Sep 10 04:51:38.996572 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.996531 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-maas-api-tls\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:38.996765 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.996599 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-metrics-certs\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:38.996765 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.996659 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9557d6f-fdc3-4260-8863-cb628ebafa62-openshift-service-ca\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:38.996765 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:38.996703 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhfz\" (UniqueName: \"kubernetes.io/projected/a9557d6f-fdc3-4260-8863-cb628ebafa62-kube-api-access-6vhfz\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:39.097371 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.097321 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-maas-api-tls\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:39.097616 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.097434 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-metrics-certs\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:39.097616 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.097496 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9557d6f-fdc3-4260-8863-cb628ebafa62-openshift-service-ca\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:39.097616 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.097525 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhfz\" (UniqueName: \"kubernetes.io/projected/a9557d6f-fdc3-4260-8863-cb628ebafa62-kube-api-access-6vhfz\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:39.098162 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.098139 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9557d6f-fdc3-4260-8863-cb628ebafa62-openshift-service-ca\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:39.100022 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.099995 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-metrics-certs\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:39.100127 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.100099 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-maas-api-tls\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:39.106433 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.106401 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhfz\" (UniqueName: \"kubernetes.io/projected/a9557d6f-fdc3-4260-8863-cb628ebafa62-kube-api-access-6vhfz\") pod \"maas-api-7ff5bf8f9d-r47g5\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:39.233800 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.233707 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:39.432006 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.431968 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4nxnk"] Sep 10 04:51:39.435753 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.435729 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" Sep 10 04:51:39.438241 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.438215 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 04:51:39.438396 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.438216 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-b6stt\"" Sep 10 04:51:39.446013 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.445980 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4nxnk"] Sep 10 04:51:39.501740 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.501705 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf246a19-caf3-4fd6-8611-add57e445231-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4nxnk\" (UID: \"cf246a19-caf3-4fd6-8611-add57e445231\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" Sep 10 04:51:39.501928 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.501787 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbj82\" (UniqueName: \"kubernetes.io/projected/cf246a19-caf3-4fd6-8611-add57e445231-kube-api-access-sbj82\") pod \"payload-processing-5b54b7fbb7-4nxnk\" (UID: \"cf246a19-caf3-4fd6-8611-add57e445231\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" Sep 10 04:51:39.511048 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.511021 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5"] Sep 10 04:51:39.513501 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:51:39.513468 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9557d6f_fdc3_4260_8863_cb628ebafa62.slice/crio-445b0378c6466a4acf4bbb8582a0690301036ea21cab960a353bf618fbaf237f WatchSource:0}: Error finding container 445b0378c6466a4acf4bbb8582a0690301036ea21cab960a353bf618fbaf237f: Status 404 returned error can't find the container with id 445b0378c6466a4acf4bbb8582a0690301036ea21cab960a353bf618fbaf237f Sep 10 04:51:39.602283 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.602247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf246a19-caf3-4fd6-8611-add57e445231-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4nxnk\" (UID: \"cf246a19-caf3-4fd6-8611-add57e445231\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" Sep 10 04:51:39.602485 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.602326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbj82\" (UniqueName: \"kubernetes.io/projected/cf246a19-caf3-4fd6-8611-add57e445231-kube-api-access-sbj82\") pod \"payload-processing-5b54b7fbb7-4nxnk\" (UID: \"cf246a19-caf3-4fd6-8611-add57e445231\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" Sep 10 04:51:39.603035 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.603002 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf246a19-caf3-4fd6-8611-add57e445231-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4nxnk\" (UID: \"cf246a19-caf3-4fd6-8611-add57e445231\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" Sep 10 04:51:39.613905 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.613871 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbj82\" (UniqueName: \"kubernetes.io/projected/cf246a19-caf3-4fd6-8611-add57e445231-kube-api-access-sbj82\") pod \"payload-processing-5b54b7fbb7-4nxnk\" (UID: \"cf246a19-caf3-4fd6-8611-add57e445231\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" Sep 10 04:51:39.748487 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:39.748424 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" Sep 10 04:51:40.023201 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:40.023093 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4nxnk"] Sep 10 04:51:40.027490 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:51:40.027158 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf246a19_caf3_4fd6_8611_add57e445231.slice/crio-001c8b853363b1a3ab6414044783942d630b7c9519dbabead53008a6b90ddda3 WatchSource:0}: Error finding container 001c8b853363b1a3ab6414044783942d630b7c9519dbabead53008a6b90ddda3: Status 404 returned error can't find the container with id 001c8b853363b1a3ab6414044783942d630b7c9519dbabead53008a6b90ddda3 Sep 10 04:51:40.199940 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:40.199907 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" event={"ID":"a9557d6f-fdc3-4260-8863-cb628ebafa62","Type":"ContainerStarted","Data":"445b0378c6466a4acf4bbb8582a0690301036ea21cab960a353bf618fbaf237f"} Sep 10 04:51:40.201038 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:40.201005 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" event={"ID":"cf246a19-caf3-4fd6-8611-add57e445231","Type":"ContainerStarted","Data":"001c8b853363b1a3ab6414044783942d630b7c9519dbabead53008a6b90ddda3"} Sep 10 04:51:43.222173 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:43.222064 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" event={"ID":"a9557d6f-fdc3-4260-8863-cb628ebafa62","Type":"ContainerStarted","Data":"1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6"} Sep 10 04:51:43.222614 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:43.222175 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:43.223670 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:43.223644 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" event={"ID":"cf246a19-caf3-4fd6-8611-add57e445231","Type":"ContainerStarted","Data":"5505a40fbbeb282392880ebe5e8130c56f709aa9e9afdab2c7406a87c7298f81"} Sep 10 04:51:43.223779 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:43.223692 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" Sep 10 04:51:43.241110 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:43.241056 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" podStartSLOduration=2.250702713 podStartE2EDuration="5.24103771s" totalImagesPullingTime="2.990334997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:51:43.23869873 +0000 UTC m=+622.742531792" watchObservedRunningTime="2026-09-10 04:51:43.24103771 +0000 UTC m=+622.744870801" Sep 10 04:51:43.254755 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:43.254694 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" podStartSLOduration=1.77898599 podStartE2EDuration="4.254676514s" totalImagesPullingTime="2.475690524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:51:43.253476187 +0000 UTC m=+622.757309265" watchObservedRunningTime="2026-09-10 04:51:43.254676514 +0000 UTC m=+622.758509574" Sep 10 04:51:49.234273 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:49.234241 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 04:51:49.257634 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:49.257560 2572 generic.go:356] "Generic (PLEG): container finished" podID="716fa502-5c07-487c-9410-4546ddfffec2" containerID="24a8f96ef336ebbb7405d3d818f79138341906f95bbb97fdd1ac09f2d3cf2f5b" exitCode=0 Sep 10 04:51:49.257823 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:49.257644 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" event={"ID":"716fa502-5c07-487c-9410-4546ddfffec2","Type":"ContainerDied","Data":"24a8f96ef336ebbb7405d3d818f79138341906f95bbb97fdd1ac09f2d3cf2f5b"} Sep 10 04:51:49.258191 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:49.258168 2572 scope.go:117] "RemoveContainer" containerID="24a8f96ef336ebbb7405d3d818f79138341906f95bbb97fdd1ac09f2d3cf2f5b" Sep 10 04:51:49.259348 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:49.259317 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:51:49.259412 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:49.259395 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:51:49.259483 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:49.259429 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:51:50.265251 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:50.265215 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" event={"ID":"716fa502-5c07-487c-9410-4546ddfffec2","Type":"ContainerStarted","Data":"b9f8612c6bcc639bfad532874fe92ac3c971085eadd7996f48c62a09f7d2157e"} Sep 10 04:51:51.038130 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:51.038090 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:51:51.038341 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:51.038306 2572 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9l5xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 10 04:51:51.038396 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:51.038353 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" podUID="716fa502-5c07-487c-9410-4546ddfffec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 10 04:51:52.037835 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:52.037797 2572 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9l5xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 10 04:51:52.038266 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:52.037866 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" podUID="716fa502-5c07-487c-9410-4546ddfffec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 10 04:51:53.037871 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:53.037832 2572 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9l5xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 10 04:51:53.038284 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:53.037889 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" podUID="716fa502-5c07-487c-9410-4546ddfffec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 10 04:51:54.038196 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:54.038158 2572 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9l5xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 10 04:51:54.038671 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:54.038219 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" podUID="716fa502-5c07-487c-9410-4546ddfffec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 10 04:51:54.231520 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:54.231482 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4nxnk" Sep 10 04:51:55.038260 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:55.038214 2572 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9l5xw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" start-of-body= Sep 10 04:51:55.038778 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:55.038299 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" podUID="716fa502-5c07-487c-9410-4546ddfffec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Sep 10 04:51:56.041379 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.041335 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:51:56.041797 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.041650 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:51:56.042504 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.042485 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9l5xw" Sep 10 04:51:56.393166 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.393008 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s"] Sep 10 04:51:56.397583 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.397545 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.400263 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.400237 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 04:51:56.400411 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.400284 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bstcg\"" Sep 10 04:51:56.400411 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.400289 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 04:51:56.400411 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.400292 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 04:51:56.407485 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.407429 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s"] Sep 10 04:51:56.469342 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.469299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.469342 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.469342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f215e50-9f97-4e49-aa61-a46a99ae38a6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.469652 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.469425 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.469652 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.469487 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.469652 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.469626 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.469766 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.469698 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrbf8\" (UniqueName: \"kubernetes.io/projected/6f215e50-9f97-4e49-aa61-a46a99ae38a6-kube-api-access-nrbf8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.570307 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.570266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.570307 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.570303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f215e50-9f97-4e49-aa61-a46a99ae38a6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.570573 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.570324 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.570573 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.570339 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.570573 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.570395 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.570573 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.570427 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrbf8\" (UniqueName: \"kubernetes.io/projected/6f215e50-9f97-4e49-aa61-a46a99ae38a6-kube-api-access-nrbf8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.570782 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.570700 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.570863 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.570843 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.570913 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.570840 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.572698 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.572675 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f215e50-9f97-4e49-aa61-a46a99ae38a6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.572975 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.572955 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f215e50-9f97-4e49-aa61-a46a99ae38a6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.578033 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.578005 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrbf8\" (UniqueName: \"kubernetes.io/projected/6f215e50-9f97-4e49-aa61-a46a99ae38a6-kube-api-access-nrbf8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s\" (UID: \"6f215e50-9f97-4e49-aa61-a46a99ae38a6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.710955 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.710858 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:51:56.987056 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.987024 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s"] Sep 10 04:51:56.990572 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:51:56.990539 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f215e50_9f97_4e49_aa61_a46a99ae38a6.slice/crio-70f7ddcb6971a04a120cfc4251ea360de7d6280dcfa6d0a1dc9f31770a2464d5 WatchSource:0}: Error finding container 70f7ddcb6971a04a120cfc4251ea360de7d6280dcfa6d0a1dc9f31770a2464d5: Status 404 returned error can't find the container with id 70f7ddcb6971a04a120cfc4251ea360de7d6280dcfa6d0a1dc9f31770a2464d5 Sep 10 04:51:56.991902 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:56.991884 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 04:51:57.317915 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:57.317817 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" event={"ID":"6f215e50-9f97-4e49-aa61-a46a99ae38a6","Type":"ContainerStarted","Data":"70f7ddcb6971a04a120cfc4251ea360de7d6280dcfa6d0a1dc9f31770a2464d5"} Sep 10 04:51:58.323747 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:58.323710 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" event={"ID":"6f215e50-9f97-4e49-aa61-a46a99ae38a6","Type":"ContainerStarted","Data":"93e43d932f8729d6f852444a3cbcf17a9f7381e5aa5b486febf66bf620cb17de"} Sep 10 04:51:58.341078 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:51:58.341024 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" podStartSLOduration=1.3494798860000001 podStartE2EDuration="2.341006599s" totalImagesPullingTime="991.526713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:51:58.339132331 +0000 UTC m=+637.842965401" watchObservedRunningTime="2026-09-10 04:51:58.341006599 +0000 UTC m=+637.844839660" Sep 10 04:52:06.711169 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:52:06.711126 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:52:06.711631 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:52:06.711187 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:52:06.723416 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:52:06.723387 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:52:07.385663 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:52:07.385633 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s" Sep 10 04:54:47.819853 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.819767 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l"] Sep 10 04:54:47.823545 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.823522 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:47.826045 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.826020 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7f2cd3-openshift-default-dockercfg-6pt2j\"" Sep 10 04:54:47.834142 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.834106 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l"] Sep 10 04:54:47.884590 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.884556 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5"] Sep 10 04:54:47.888189 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.888167 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:47.890801 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.890777 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-13af27-openshift-default-dockercfg-ldmjv\"" Sep 10 04:54:47.900022 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.899993 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5"] Sep 10 04:54:47.919433 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.919397 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-credential-socket\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:47.919433 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.919433 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-envoy\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:47.919669 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.919547 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-data\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:47.919669 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.919591 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-token\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:47.919745 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.919667 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-certs\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:47.919745 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.919708 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-socket\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:47.919745 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.919736 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-podinfo\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:47.919863 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.919825 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84205a9-1655-4123-9b56-c1f235f11ddb-istiod-ca-cert\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:47.919923 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:47.919872 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsfhj\" (UniqueName: \"kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-kube-api-access-tsfhj\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021081 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021044 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-certs\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021288 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-credential-socket\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.021288 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-certs\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.021288 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021246 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da54c73-ad57-4436-8b1f-0f421816feff-istio-podinfo\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.021288 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021267 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-istio-token\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.021542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021299 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-socket\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-podinfo\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021350 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgff4\" (UniqueName: \"kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-kube-api-access-dgff4\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.021542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84205a9-1655-4123-9b56-c1f235f11ddb-istiod-ca-cert\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021432 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-socket\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.021542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021493 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsfhj\" (UniqueName: \"kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-kube-api-access-tsfhj\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021522 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-data\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.021542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021540 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-certs\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021946 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021626 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-credential-socket\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021946 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021624 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-socket\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021946 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-envoy\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021946 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021681 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-data\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021946 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021707 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-envoy\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.021946 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021736 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-token\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.021946 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021774 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da54c73-ad57-4436-8b1f-0f421816feff-istiod-ca-cert\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.022209 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.021966 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-credential-socket\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.022209 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.022025 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-data\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.022209 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.022187 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84205a9-1655-4123-9b56-c1f235f11ddb-istiod-ca-cert\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.023975 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.023951 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-envoy\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.024268 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.024248 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-podinfo\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.029371 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.029337 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-token\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.029808 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.029789 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsfhj\" (UniqueName: \"kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-kube-api-access-tsfhj\") pod \"e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.123489 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.123291 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da54c73-ad57-4436-8b1f-0f421816feff-istiod-ca-cert\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.123489 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.123370 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-credential-socket\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.123489 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.123395 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-certs\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.123489 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.123422 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da54c73-ad57-4436-8b1f-0f421816feff-istio-podinfo\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.123489 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.123464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-istio-token\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.123931 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.123604 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgff4\" (UniqueName: \"kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-kube-api-access-dgff4\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.123931 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.123688 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-socket\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.123931 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.123754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-data\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.123931 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.123813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-envoy\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.123931 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.123907 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-certs\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.124186 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.124079 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-socket\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.124302 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.124260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-data\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.124652 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.124627 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-credential-socket\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.124853 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.124832 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da54c73-ad57-4436-8b1f-0f421816feff-istiod-ca-cert\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.126840 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.126814 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-envoy\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.127183 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.127160 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da54c73-ad57-4436-8b1f-0f421816feff-istio-podinfo\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.131033 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.131006 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-istio-token\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.131217 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.131192 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgff4\" (UniqueName: \"kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-kube-api-access-dgff4\") pod \"e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.135017 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.134987 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:48.200960 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.200918 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:48.413103 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.413074 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l"] Sep 10 04:54:48.415596 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:54:48.415567 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84205a9_1655_4123_9b56_c1f235f11ddb.slice/crio-db5a950bc7307c9293516a5df6c233fb5aa0f011df4a9e6d104904f1e6f4acbd WatchSource:0}: Error finding container db5a950bc7307c9293516a5df6c233fb5aa0f011df4a9e6d104904f1e6f4acbd: Status 404 returned error can't find the container with id db5a950bc7307c9293516a5df6c233fb5aa0f011df4a9e6d104904f1e6f4acbd Sep 10 04:54:48.417840 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.417806 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:54:48.417939 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.417891 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:54:48.417939 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.417934 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:54:48.487040 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.487007 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5"] Sep 10 04:54:48.490290 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:54:48.490264 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da54c73_ad57_4436_8b1f_0f421816feff.slice/crio-35043f3a14b60b3dbeabd8687d51de3c707bd68ebd92cf8ce6254aa889291967 WatchSource:0}: Error finding container 35043f3a14b60b3dbeabd8687d51de3c707bd68ebd92cf8ce6254aa889291967: Status 404 returned error can't find the container with id 35043f3a14b60b3dbeabd8687d51de3c707bd68ebd92cf8ce6254aa889291967 Sep 10 04:54:48.493258 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.493222 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:54:48.493339 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.493304 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:54:48.493379 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:48.493358 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:54:49.178661 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:49.178616 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" event={"ID":"8da54c73-ad57-4436-8b1f-0f421816feff","Type":"ContainerStarted","Data":"1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4"} Sep 10 04:54:49.179137 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:49.178679 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" event={"ID":"8da54c73-ad57-4436-8b1f-0f421816feff","Type":"ContainerStarted","Data":"35043f3a14b60b3dbeabd8687d51de3c707bd68ebd92cf8ce6254aa889291967"} Sep 10 04:54:49.180300 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:49.180274 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" event={"ID":"d84205a9-1655-4123-9b56-c1f235f11ddb","Type":"ContainerStarted","Data":"ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0"} Sep 10 04:54:49.180400 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:49.180323 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" event={"ID":"d84205a9-1655-4123-9b56-c1f235f11ddb","Type":"ContainerStarted","Data":"db5a950bc7307c9293516a5df6c233fb5aa0f011df4a9e6d104904f1e6f4acbd"} Sep 10 04:54:49.201536 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:49.201497 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:49.202735 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:49.202689 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" podStartSLOduration=2.202673545 podStartE2EDuration="2.202673545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:49.201167629 +0000 UTC m=+808.705000693" watchObservedRunningTime="2026-09-10 04:54:49.202673545 +0000 UTC m=+808.706506606" Sep 10 04:54:49.207043 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:49.207017 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:49.222761 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:49.222710 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" podStartSLOduration=2.2226932440000002 podStartE2EDuration="2.222693244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:49.220844259 +0000 UTC m=+808.724677321" watchObservedRunningTime="2026-09-10 04:54:49.222693244 +0000 UTC m=+808.726526305" Sep 10 04:54:50.135414 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:50.135378 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:50.140537 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:50.140507 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:50.185998 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:50.185919 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:50.185998 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:50.185975 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:50.186751 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:50.186720 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:54:50.186856 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:50.186774 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:54:52.328500 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.328464 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x"] Sep 10 04:54:52.341609 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.341262 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.343914 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.343885 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-7f2cd3-serving-cert\"" Sep 10 04:54:52.344084 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.344055 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x"] Sep 10 04:54:52.367931 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.367891 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.368132 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.367962 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k26q9\" (UniqueName: \"kubernetes.io/projected/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-kube-api-access-k26q9\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.368132 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.368019 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.368132 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.368116 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-metrics-certs\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.469244 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.469207 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.469441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.469261 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k26q9\" (UniqueName: \"kubernetes.io/projected/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-kube-api-access-k26q9\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.469441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.469311 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.469441 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:54:52.469383 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-serving-cert: secret "maas-api-e2e-worker-w0-7f2cd3-serving-cert" not found Sep 10 04:54:52.469654 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.469492 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-metrics-certs\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.469654 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:54:52.469502 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-maas-api-tls podName:255a1ddc-1b5e-491d-98c0-fcfdac6233c1 nodeName:}" failed. No retries permitted until 2026-09-10 04:54:52.969444293 +0000 UTC m=+812.473277351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-maas-api-tls") pod "maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" (UID: "255a1ddc-1b5e-491d-98c0-fcfdac6233c1") : secret "maas-api-e2e-worker-w0-7f2cd3-serving-cert" not found Sep 10 04:54:52.470079 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.470059 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.471862 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.471842 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-metrics-certs\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.481743 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.481715 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k26q9\" (UniqueName: \"kubernetes.io/projected/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-kube-api-access-k26q9\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.975482 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.975408 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:52.977828 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:52.977802 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:53.156198 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.156153 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz"] Sep 10 04:54:53.160674 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.160645 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 04:54:53.163211 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.163180 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7f2cd3\"" Sep 10 04:54:53.163478 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.163279 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-7f2cd3-dockercfg-fcghx\"" Sep 10 04:54:53.183287 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.183086 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l72ww\" (UniqueName: \"kubernetes.io/projected/6217e20c-03ca-4fa9-b9cf-457b28c35f30-kube-api-access-l72ww\") pod \"payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz\" (UID: \"6217e20c-03ca-4fa9-b9cf-457b28c35f30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 04:54:53.183470 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.183367 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6217e20c-03ca-4fa9-b9cf-457b28c35f30-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz\" (UID: \"6217e20c-03ca-4fa9-b9cf-457b28c35f30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 04:54:53.193941 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.193901 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz"] Sep 10 04:54:53.254067 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.254016 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:53.285978 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.285939 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l72ww\" (UniqueName: \"kubernetes.io/projected/6217e20c-03ca-4fa9-b9cf-457b28c35f30-kube-api-access-l72ww\") pod \"payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz\" (UID: \"6217e20c-03ca-4fa9-b9cf-457b28c35f30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 04:54:53.286162 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.286002 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6217e20c-03ca-4fa9-b9cf-457b28c35f30-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz\" (UID: \"6217e20c-03ca-4fa9-b9cf-457b28c35f30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 04:54:53.286822 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.286797 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6217e20c-03ca-4fa9-b9cf-457b28c35f30-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz\" (UID: \"6217e20c-03ca-4fa9-b9cf-457b28c35f30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 04:54:53.309820 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.309786 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l72ww\" (UniqueName: \"kubernetes.io/projected/6217e20c-03ca-4fa9-b9cf-457b28c35f30-kube-api-access-l72ww\") pod \"payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz\" (UID: \"6217e20c-03ca-4fa9-b9cf-457b28c35f30\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 04:54:53.344849 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.344609 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q"] Sep 10 04:54:53.350312 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.350279 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.353892 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.353649 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-13af27-serving-cert\"" Sep 10 04:54:53.362959 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.361106 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q"] Sep 10 04:54:53.386608 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.386572 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.386814 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.386654 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c226b3-d201-4323-9c0d-3714ba0b672a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.386814 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.386692 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9rq\" (UniqueName: \"kubernetes.io/projected/01c226b3-d201-4323-9c0d-3714ba0b672a-kube-api-access-dw9rq\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.386915 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.386804 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-metrics-certs\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.487880 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.487843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-metrics-certs\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.488047 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.487918 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.488047 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.487957 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c226b3-d201-4323-9c0d-3714ba0b672a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.488047 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.487981 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9rq\" (UniqueName: \"kubernetes.io/projected/01c226b3-d201-4323-9c0d-3714ba0b672a-kube-api-access-dw9rq\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.488226 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:54:53.488068 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-serving-cert: secret "maas-api-e2e-worker-w4-13af27-serving-cert" not found Sep 10 04:54:53.488226 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:54:53.488157 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-maas-api-tls podName:01c226b3-d201-4323-9c0d-3714ba0b672a nodeName:}" failed. No retries permitted until 2026-09-10 04:54:53.988138496 +0000 UTC m=+813.491971535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-maas-api-tls") pod "maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" (UID: "01c226b3-d201-4323-9c0d-3714ba0b672a") : secret "maas-api-e2e-worker-w4-13af27-serving-cert" not found Sep 10 04:54:53.488775 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.488754 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c226b3-d201-4323-9c0d-3714ba0b672a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.490405 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.490379 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-metrics-certs\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.490568 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.490553 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 04:54:53.503045 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.503017 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9rq\" (UniqueName: \"kubernetes.io/projected/01c226b3-d201-4323-9c0d-3714ba0b672a-kube-api-access-dw9rq\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.535896 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.535869 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x"] Sep 10 04:54:53.537775 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:54:53.537749 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255a1ddc_1b5e_491d_98c0_fcfdac6233c1.slice/crio-04a83823fd07b6f7588831aea0ecc46a9eb1fc9e3b624fb9c506bb70ce8b4137 WatchSource:0}: Error finding container 04a83823fd07b6f7588831aea0ecc46a9eb1fc9e3b624fb9c506bb70ce8b4137: Status 404 returned error can't find the container with id 04a83823fd07b6f7588831aea0ecc46a9eb1fc9e3b624fb9c506bb70ce8b4137 Sep 10 04:54:53.779593 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.776999 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz"] Sep 10 04:54:53.779593 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:54:53.779400 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6217e20c_03ca_4fa9_b9cf_457b28c35f30.slice/crio-ac8e49e45f5b9e7479fd411c6c713b715369d6415e7ec4b7403f72ae1f8d59fb WatchSource:0}: Error finding container ac8e49e45f5b9e7479fd411c6c713b715369d6415e7ec4b7403f72ae1f8d59fb: Status 404 returned error can't find the container with id ac8e49e45f5b9e7479fd411c6c713b715369d6415e7ec4b7403f72ae1f8d59fb Sep 10 04:54:53.994984 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.994947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:53.998106 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:53.998072 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:54.217663 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:54.217547 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" event={"ID":"255a1ddc-1b5e-491d-98c0-fcfdac6233c1","Type":"ContainerStarted","Data":"3f1d32e0479c2d2839dfe1aeafc5a11ee4bdcfa9642dcd95ee00d26188e5d7ea"} Sep 10 04:54:54.217663 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:54.217625 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" event={"ID":"255a1ddc-1b5e-491d-98c0-fcfdac6233c1","Type":"ContainerStarted","Data":"04a83823fd07b6f7588831aea0ecc46a9eb1fc9e3b624fb9c506bb70ce8b4137"} Sep 10 04:54:54.217904 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:54.217757 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:54:54.219253 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:54.219226 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" event={"ID":"6217e20c-03ca-4fa9-b9cf-457b28c35f30","Type":"ContainerStarted","Data":"af147d6d5f3e11d9a4363b6b24e1b9ef10f7e3329743179ff310feb6178faa52"} Sep 10 04:54:54.219391 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:54.219272 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" event={"ID":"6217e20c-03ca-4fa9-b9cf-457b28c35f30","Type":"ContainerStarted","Data":"ac8e49e45f5b9e7479fd411c6c713b715369d6415e7ec4b7403f72ae1f8d59fb"} Sep 10 04:54:54.219391 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:54.219359 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 04:54:54.236618 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:54.236555 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" podStartSLOduration=1.85631184 podStartE2EDuration="2.23653268s" totalImagesPullingTime="380.22084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:54.234228682 +0000 UTC m=+813.738061755" watchObservedRunningTime="2026-09-10 04:54:54.23653268 +0000 UTC m=+813.740365744" Sep 10 04:54:54.253296 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:54.252554 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" podStartSLOduration=1.252529043 podStartE2EDuration="1.252529043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:54.251397388 +0000 UTC m=+813.755230452" watchObservedRunningTime="2026-09-10 04:54:54.252529043 +0000 UTC m=+813.756362105" Sep 10 04:54:54.269238 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:54.269200 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:54.551265 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:54.551069 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q"] Sep 10 04:54:54.554068 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:54:54.554038 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c226b3_d201_4323_9c0d_3714ba0b672a.slice/crio-f50ea44d2d5e078d04d55efe930da5d2705354c007e9ebcb4e4e5d28f68169ad WatchSource:0}: Error finding container f50ea44d2d5e078d04d55efe930da5d2705354c007e9ebcb4e4e5d28f68169ad: Status 404 returned error can't find the container with id f50ea44d2d5e078d04d55efe930da5d2705354c007e9ebcb4e4e5d28f68169ad Sep 10 04:54:55.225860 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:55.225767 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" event={"ID":"01c226b3-d201-4323-9c0d-3714ba0b672a","Type":"ContainerStarted","Data":"fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696"} Sep 10 04:54:55.225860 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:55.225820 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" event={"ID":"01c226b3-d201-4323-9c0d-3714ba0b672a","Type":"ContainerStarted","Data":"f50ea44d2d5e078d04d55efe930da5d2705354c007e9ebcb4e4e5d28f68169ad"} Sep 10 04:54:55.226099 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:55.226014 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:54:55.244628 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:55.244576 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" podStartSLOduration=1.856532555 podStartE2EDuration="2.244556779s" totalImagesPullingTime="388.024224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:55.243302933 +0000 UTC m=+814.747135996" watchObservedRunningTime="2026-09-10 04:54:55.244556779 +0000 UTC m=+814.748389839" Sep 10 04:54:56.512192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.512154 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t"] Sep 10 04:54:56.516908 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.516874 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.519633 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.519603 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e79d9e97-openshift-default-dockercfg-p6jgz\"" Sep 10 04:54:56.519877 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.519764 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t"] Sep 10 04:54:56.626837 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.626792 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xxf\" (UniqueName: \"kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-kube-api-access-p4xxf\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.627008 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.626852 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9076fab0-fd5c-4195-8f21-89332de59d84-istiod-ca-cert\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.627008 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.626887 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-data\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.627008 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.626920 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-credential-socket\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.627008 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.626960 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-istio-token\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.627008 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.626976 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-socket\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.627182 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.627039 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9076fab0-fd5c-4195-8f21-89332de59d84-istio-podinfo\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.627182 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.627078 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-certs\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.627182 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.627094 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-envoy\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.728400 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.728360 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-istio-token\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.728400 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.728403 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-socket\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.728670 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.728600 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9076fab0-fd5c-4195-8f21-89332de59d84-istio-podinfo\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.728774 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.728750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-certs\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.728816 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.728782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-envoy\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.728816 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.728805 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-socket\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.728887 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.728838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xxf\" (UniqueName: \"kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-kube-api-access-p4xxf\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.728887 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.728881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9076fab0-fd5c-4195-8f21-89332de59d84-istiod-ca-cert\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.728971 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.728904 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-data\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.728971 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.728940 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-credential-socket\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.729219 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.729190 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-certs\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.729308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.729270 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-data\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.729544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.729525 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-credential-socket\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.729763 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.729739 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9076fab0-fd5c-4195-8f21-89332de59d84-istiod-ca-cert\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.731131 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.731111 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-envoy\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.731354 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.731336 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9076fab0-fd5c-4195-8f21-89332de59d84-istio-podinfo\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.743359 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.743327 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-istio-token\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.743775 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.743747 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xxf\" (UniqueName: \"kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-kube-api-access-p4xxf\") pod \"e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:56.830964 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:56.830813 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:57.124950 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:57.124923 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t"] Sep 10 04:54:57.126864 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:54:57.126841 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9076fab0_fd5c_4195_8f21_89332de59d84.slice/crio-411a0d5818054e180e72ec41a1414167a8fbb2a8e5d4c8fc6834314779efb64e WatchSource:0}: Error finding container 411a0d5818054e180e72ec41a1414167a8fbb2a8e5d4c8fc6834314779efb64e: Status 404 returned error can't find the container with id 411a0d5818054e180e72ec41a1414167a8fbb2a8e5d4c8fc6834314779efb64e Sep 10 04:54:57.129159 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:57.129127 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:54:57.129240 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:57.129195 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:54:57.129240 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:57.129223 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:54:57.239579 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:57.239545 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" event={"ID":"9076fab0-fd5c-4195-8f21-89332de59d84","Type":"ContainerStarted","Data":"7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b"} Sep 10 04:54:57.239782 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:57.239611 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" event={"ID":"9076fab0-fd5c-4195-8f21-89332de59d84","Type":"ContainerStarted","Data":"411a0d5818054e180e72ec41a1414167a8fbb2a8e5d4c8fc6834314779efb64e"} Sep 10 04:54:57.263964 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:57.263908 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" podStartSLOduration=1.263890421 podStartE2EDuration="1.263890421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:54:57.262061916 +0000 UTC m=+816.765895011" watchObservedRunningTime="2026-09-10 04:54:57.263890421 +0000 UTC m=+816.767723481" Sep 10 04:54:57.831260 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:57.831221 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:58.836523 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:58.836487 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:59.250584 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:59.250551 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:59.251651 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:59.251631 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:54:59.991379 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:59.991341 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x"] Sep 10 04:54:59.995265 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:59.995234 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:54:59.997899 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:54:59.997874 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e79d9e97-serving-cert\"" Sep 10 04:55:00.005618 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.005573 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x"] Sep 10 04:55:00.064268 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.064217 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-metrics-certs\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.064492 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.064296 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvzrl\" (UniqueName: \"kubernetes.io/projected/087ee327-2005-4005-a929-865122a9d431-kube-api-access-xvzrl\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.064492 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.064381 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087ee327-2005-4005-a929-865122a9d431-openshift-service-ca\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.064492 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.064407 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-maas-api-tls\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.165878 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.165830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-metrics-certs\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.166082 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.165926 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvzrl\" (UniqueName: \"kubernetes.io/projected/087ee327-2005-4005-a929-865122a9d431-kube-api-access-xvzrl\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.166082 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.165975 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087ee327-2005-4005-a929-865122a9d431-openshift-service-ca\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.166082 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.165994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-maas-api-tls\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.166265 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:55:00.166125 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-serving-cert: secret "maas-api-e2e-ait-e79d9e97-serving-cert" not found Sep 10 04:55:00.166265 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:55:00.166201 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-maas-api-tls podName:087ee327-2005-4005-a929-865122a9d431 nodeName:}" failed. No retries permitted until 2026-09-10 04:55:00.66617799 +0000 UTC m=+820.170011031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-maas-api-tls") pod "maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" (UID: "087ee327-2005-4005-a929-865122a9d431") : secret "maas-api-e2e-ait-e79d9e97-serving-cert" not found Sep 10 04:55:00.166758 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.166736 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087ee327-2005-4005-a929-865122a9d431-openshift-service-ca\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.168499 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.168474 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-metrics-certs\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.175465 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.175414 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvzrl\" (UniqueName: \"kubernetes.io/projected/087ee327-2005-4005-a929-865122a9d431-kube-api-access-xvzrl\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.233431 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.233390 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 04:55:00.671021 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.670977 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-maas-api-tls\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.673456 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.673420 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-maas-api-tls\") pod \"maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:00.911652 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:00.911613 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:01.191175 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:01.191075 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x"] Sep 10 04:55:01.195166 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:55:01.195134 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087ee327_2005_4005_a929_865122a9d431.slice/crio-e01f71f34fdfed17f8e8ad1e79d14c17cfd8ba44924e24155b273891944cb87e WatchSource:0}: Error finding container e01f71f34fdfed17f8e8ad1e79d14c17cfd8ba44924e24155b273891944cb87e: Status 404 returned error can't find the container with id e01f71f34fdfed17f8e8ad1e79d14c17cfd8ba44924e24155b273891944cb87e Sep 10 04:55:01.238546 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:01.238515 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 04:55:01.271563 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:01.271427 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" event={"ID":"087ee327-2005-4005-a929-865122a9d431","Type":"ContainerStarted","Data":"e01f71f34fdfed17f8e8ad1e79d14c17cfd8ba44924e24155b273891944cb87e"} Sep 10 04:55:02.280487 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:02.280429 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" event={"ID":"087ee327-2005-4005-a929-865122a9d431","Type":"ContainerStarted","Data":"5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8"} Sep 10 04:55:02.281360 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:02.281327 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:02.303475 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:02.302076 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" podStartSLOduration=2.642183074 podStartE2EDuration="3.302051148s" totalImagesPullingTime="659.868074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:55:02.299109677 +0000 UTC m=+821.802942740" watchObservedRunningTime="2026-09-10 04:55:02.302051148 +0000 UTC m=+821.805884210" Sep 10 04:55:03.151973 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.151939 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk"] Sep 10 04:55:03.155928 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.155908 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:55:03.158438 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.158279 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-53c734\"" Sep 10 04:55:03.164363 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.164321 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk"] Sep 10 04:55:03.193372 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.190156 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb"] Sep 10 04:55:03.203474 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.197756 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:55:03.203474 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.202476 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-53c734-dockercfg-hpnkz\"" Sep 10 04:55:03.205678 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.205645 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb"] Sep 10 04:55:03.324475 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.307549 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebfb768-521c-4c7a-aa3f-d526248d437f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk\" (UID: \"bebfb768-521c-4c7a-aa3f-d526248d437f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:55:03.324475 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.307624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-plugins-config-volume\") pod \"payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb\" (UID: \"bdd993d4-cf3c-463c-9c34-0c223b74ae7a\") " pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:55:03.324475 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.307726 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndqv8\" (UniqueName: \"kubernetes.io/projected/bebfb768-521c-4c7a-aa3f-d526248d437f-kube-api-access-ndqv8\") pod \"payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk\" (UID: \"bebfb768-521c-4c7a-aa3f-d526248d437f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:55:03.324475 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.307767 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc5nd\" (UniqueName: \"kubernetes.io/projected/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-kube-api-access-sc5nd\") pod \"payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb\" (UID: \"bdd993d4-cf3c-463c-9c34-0c223b74ae7a\") " pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:55:03.412526 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.408913 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndqv8\" (UniqueName: \"kubernetes.io/projected/bebfb768-521c-4c7a-aa3f-d526248d437f-kube-api-access-ndqv8\") pod \"payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk\" (UID: \"bebfb768-521c-4c7a-aa3f-d526248d437f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:55:03.412526 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.408990 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc5nd\" (UniqueName: \"kubernetes.io/projected/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-kube-api-access-sc5nd\") pod \"payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb\" (UID: \"bdd993d4-cf3c-463c-9c34-0c223b74ae7a\") " pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:55:03.412526 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.409898 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebfb768-521c-4c7a-aa3f-d526248d437f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk\" (UID: \"bebfb768-521c-4c7a-aa3f-d526248d437f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:55:03.412526 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.409944 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-plugins-config-volume\") pod \"payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb\" (UID: \"bdd993d4-cf3c-463c-9c34-0c223b74ae7a\") " pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:55:03.412526 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.411173 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebfb768-521c-4c7a-aa3f-d526248d437f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk\" (UID: \"bebfb768-521c-4c7a-aa3f-d526248d437f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:55:03.412526 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.411819 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-plugins-config-volume\") pod \"payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb\" (UID: \"bdd993d4-cf3c-463c-9c34-0c223b74ae7a\") " pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:55:03.420474 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.419063 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc5nd\" (UniqueName: \"kubernetes.io/projected/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-kube-api-access-sc5nd\") pod \"payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb\" (UID: \"bdd993d4-cf3c-463c-9c34-0c223b74ae7a\") " pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:55:03.420474 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.419688 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndqv8\" (UniqueName: \"kubernetes.io/projected/bebfb768-521c-4c7a-aa3f-d526248d437f-kube-api-access-ndqv8\") pod \"payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk\" (UID: \"bebfb768-521c-4c7a-aa3f-d526248d437f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:55:03.467593 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.467551 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:55:03.540797 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.540751 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:55:03.789516 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.784605 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk"] Sep 10 04:55:03.884473 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:03.882738 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb"] Sep 10 04:55:04.355482 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:04.355221 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" event={"ID":"bdd993d4-cf3c-463c-9c34-0c223b74ae7a","Type":"ContainerStarted","Data":"9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb"} Sep 10 04:55:04.355482 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:04.355285 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" event={"ID":"bdd993d4-cf3c-463c-9c34-0c223b74ae7a","Type":"ContainerStarted","Data":"554014ea733f6eb668a4029be609809e79d846b46e32ae6da5d25bfc0512daf3"} Sep 10 04:55:04.355482 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:04.355337 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:55:04.356829 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:04.356799 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" event={"ID":"bebfb768-521c-4c7a-aa3f-d526248d437f","Type":"ContainerStarted","Data":"012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34"} Sep 10 04:55:04.356939 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:04.356849 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" event={"ID":"bebfb768-521c-4c7a-aa3f-d526248d437f","Type":"ContainerStarted","Data":"07b218a2a2f133833103f2442131adf159af0fdb5372b1785e9659571b8d8f16"} Sep 10 04:55:04.357036 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:04.357020 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:55:04.391583 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:04.391519 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" podStartSLOduration=1.391494545 podStartE2EDuration="1.391494545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:55:04.374018437 +0000 UTC m=+823.877851521" watchObservedRunningTime="2026-09-10 04:55:04.391494545 +0000 UTC m=+823.895327610" Sep 10 04:55:04.394006 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:04.393941 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" podStartSLOduration=1.3939227920000001 podStartE2EDuration="1.393922792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:55:04.392024986 +0000 UTC m=+823.895858074" watchObservedRunningTime="2026-09-10 04:55:04.393922792 +0000 UTC m=+823.897755855" Sep 10 04:55:05.228345 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:05.228308 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 04:55:09.363752 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:09.363720 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:11.413549 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.413510 2572 generic.go:356] "Generic (PLEG): container finished" podID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerID="ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0" exitCode=0 Sep 10 04:55:11.414054 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.413601 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" event={"ID":"d84205a9-1655-4123-9b56-c1f235f11ddb","Type":"ContainerDied","Data":"ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0"} Sep 10 04:55:11.414167 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.414107 2572 scope.go:117] "RemoveContainer" containerID="ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0" Sep 10 04:55:11.415508 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.415486 2572 generic.go:356] "Generic (PLEG): container finished" podID="8da54c73-ad57-4436-8b1f-0f421816feff" containerID="1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4" exitCode=0 Sep 10 04:55:11.415621 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.415512 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:55:11.415621 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.415550 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" event={"ID":"8da54c73-ad57-4436-8b1f-0f421816feff","Type":"ContainerDied","Data":"1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4"} Sep 10 04:55:11.415621 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.415583 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:55:11.415621 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.415619 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:55:11.415991 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.415974 2572 scope.go:117] "RemoveContainer" containerID="1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4" Sep 10 04:55:11.417181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.417084 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:55:11.417181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.417145 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:55:11.417181 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:11.417179 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:55:12.427193 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:12.427141 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" event={"ID":"d84205a9-1655-4123-9b56-c1f235f11ddb","Type":"ContainerStarted","Data":"64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98"} Sep 10 04:55:12.434650 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:12.434608 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" event={"ID":"8da54c73-ad57-4436-8b1f-0f421816feff","Type":"ContainerStarted","Data":"cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8"} Sep 10 04:55:13.135557 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:13.135499 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:55:13.135793 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:13.135643 2572 patch_prober.go:28] interesting pod/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 04:55:13.135793 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:13.135692 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 04:55:13.202516 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:13.202479 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:55:13.202910 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:13.202821 2572 patch_prober.go:28] interesting pod/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Sep 10 04:55:13.203079 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:13.203057 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Sep 10 04:55:14.136441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:14.136397 2572 patch_prober.go:28] interesting pod/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 04:55:14.137064 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:14.137030 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 04:55:14.202368 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:14.202326 2572 patch_prober.go:28] interesting pod/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Sep 10 04:55:14.202688 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:14.202653 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Sep 10 04:55:15.136037 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:15.135984 2572 patch_prober.go:28] interesting pod/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 04:55:15.136254 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:15.136059 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 04:55:15.203097 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:15.202631 2572 patch_prober.go:28] interesting pod/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Sep 10 04:55:15.203097 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:15.202702 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Sep 10 04:55:15.368487 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:15.368428 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:55:15.368912 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:15.368885 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:55:16.135869 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:16.135811 2572 patch_prober.go:28] interesting pod/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 04:55:16.136192 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:16.136156 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 04:55:16.202646 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:16.202595 2572 patch_prober.go:28] interesting pod/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Sep 10 04:55:16.203018 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:16.202978 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Sep 10 04:55:17.136423 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:17.136373 2572 patch_prober.go:28] interesting pod/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Sep 10 04:55:17.136952 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:17.136470 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Sep 10 04:55:17.201807 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:17.201758 2572 patch_prober.go:28] interesting pod/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Sep 10 04:55:17.202011 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:17.201830 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Sep 10 04:55:18.245910 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:18.245871 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:55:18.246828 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:18.246802 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:55:18.247573 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:18.247542 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 04:55:18.493000 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:18.492964 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:55:18.493634 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:18.493600 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:55:18.495099 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:18.495076 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 04:55:21.573964 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:21.572625 2572 generic.go:356] "Generic (PLEG): container finished" podID="9076fab0-fd5c-4195-8f21-89332de59d84" containerID="7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b" exitCode=0 Sep 10 04:55:21.573964 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:21.572707 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" event={"ID":"9076fab0-fd5c-4195-8f21-89332de59d84","Type":"ContainerDied","Data":"7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b"} Sep 10 04:55:21.573964 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:21.573237 2572 scope.go:117] "RemoveContainer" containerID="7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b" Sep 10 04:55:21.575244 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:21.575175 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:55:21.575390 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:21.575308 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:55:21.575390 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:21.575355 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:55:22.588957 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:22.588911 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" event={"ID":"9076fab0-fd5c-4195-8f21-89332de59d84","Type":"ContainerStarted","Data":"c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781"} Sep 10 04:55:22.831901 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:22.831856 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:55:22.832142 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:22.832117 2572 patch_prober.go:28] interesting pod/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 10 04:55:22.832232 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:22.832191 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 10 04:55:23.831926 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:23.831885 2572 patch_prober.go:28] interesting pod/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 10 04:55:23.832315 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:23.831947 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 10 04:55:24.831290 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:24.831248 2572 patch_prober.go:28] interesting pod/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 10 04:55:24.831536 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:24.831315 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 10 04:55:25.832088 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:25.832041 2572 patch_prober.go:28] interesting pod/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 10 04:55:25.832523 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:25.832116 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 10 04:55:26.831877 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:26.831834 2572 patch_prober.go:28] interesting pod/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 10 04:55:26.832077 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:26.831900 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 10 04:55:27.831695 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:27.831655 2572 patch_prober.go:28] interesting pod/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Sep 10 04:55:27.832117 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:27.831730 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Sep 10 04:55:28.835780 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:28.835746 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:55:28.836242 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:28.836049 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:55:28.836908 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:28.836887 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:55:40.653325 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:40.653292 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x"] Sep 10 04:55:40.653770 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:40.653522 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" podUID="087ee327-2005-4005-a929-865122a9d431" containerName="maas-api" containerID="cri-o://5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8" gracePeriod=30 Sep 10 04:55:40.919198 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:40.919125 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:40.988567 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:40.988529 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t"] Sep 10 04:55:40.989076 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:40.988804 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" containerID="cri-o://c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781" gracePeriod=30 Sep 10 04:55:41.054533 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.054490 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087ee327-2005-4005-a929-865122a9d431-openshift-service-ca\") pod \"087ee327-2005-4005-a929-865122a9d431\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " Sep 10 04:55:41.054708 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.054582 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-metrics-certs\") pod \"087ee327-2005-4005-a929-865122a9d431\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " Sep 10 04:55:41.054708 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.054605 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvzrl\" (UniqueName: \"kubernetes.io/projected/087ee327-2005-4005-a929-865122a9d431-kube-api-access-xvzrl\") pod \"087ee327-2005-4005-a929-865122a9d431\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " Sep 10 04:55:41.054708 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.054633 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-maas-api-tls\") pod \"087ee327-2005-4005-a929-865122a9d431\" (UID: \"087ee327-2005-4005-a929-865122a9d431\") " Sep 10 04:55:41.054920 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.054884 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087ee327-2005-4005-a929-865122a9d431-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "087ee327-2005-4005-a929-865122a9d431" (UID: "087ee327-2005-4005-a929-865122a9d431"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:55:41.055131 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.055110 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/087ee327-2005-4005-a929-865122a9d431-openshift-service-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:41.057004 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.056976 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087ee327-2005-4005-a929-865122a9d431-kube-api-access-xvzrl" (OuterVolumeSpecName: "kube-api-access-xvzrl") pod "087ee327-2005-4005-a929-865122a9d431" (UID: "087ee327-2005-4005-a929-865122a9d431"). InnerVolumeSpecName "kube-api-access-xvzrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:55:41.057004 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.056983 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "087ee327-2005-4005-a929-865122a9d431" (UID: "087ee327-2005-4005-a929-865122a9d431"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:55:41.057159 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.057003 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "087ee327-2005-4005-a929-865122a9d431" (UID: "087ee327-2005-4005-a929-865122a9d431"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:55:41.156396 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.156352 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:41.156396 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.156397 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvzrl\" (UniqueName: \"kubernetes.io/projected/087ee327-2005-4005-a929-865122a9d431-kube-api-access-xvzrl\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:41.156720 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.156412 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/087ee327-2005-4005-a929-865122a9d431-maas-api-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:41.698094 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.698059 2572 generic.go:356] "Generic (PLEG): container finished" podID="087ee327-2005-4005-a929-865122a9d431" containerID="5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8" exitCode=0 Sep 10 04:55:41.698604 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.698110 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" Sep 10 04:55:41.698604 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.698149 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" event={"ID":"087ee327-2005-4005-a929-865122a9d431","Type":"ContainerDied","Data":"5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8"} Sep 10 04:55:41.698604 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.698203 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x" event={"ID":"087ee327-2005-4005-a929-865122a9d431","Type":"ContainerDied","Data":"e01f71f34fdfed17f8e8ad1e79d14c17cfd8ba44924e24155b273891944cb87e"} Sep 10 04:55:41.698604 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.698225 2572 scope.go:117] "RemoveContainer" containerID="5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8" Sep 10 04:55:41.709767 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.709603 2572 scope.go:117] "RemoveContainer" containerID="5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8" Sep 10 04:55:41.710341 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:55:41.710308 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8\": container with ID starting with 5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8 not found: ID does not exist" containerID="5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8" Sep 10 04:55:41.710485 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.710349 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8"} err="failed to get container status \"5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8\": rpc error: code = NotFound desc = could not find container \"5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8\": container with ID starting with 5cc38bb0b6980ab67dc236517641c27dd0baaceac2a04e7a40229e9f195a20b8 not found: ID does not exist" Sep 10 04:55:41.721266 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.721230 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x"] Sep 10 04:55:41.726652 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:41.726623 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e79d9e97-9f6954f4b-b598x"] Sep 10 04:55:43.037788 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:43.037748 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="087ee327-2005-4005-a929-865122a9d431" path="/var/lib/kubelet/pods/087ee327-2005-4005-a929-865122a9d431/volumes" Sep 10 04:55:43.838036 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:43.837997 2572 patch_prober.go:28] interesting pod/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 04:55:43.838217 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:43.838067 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 04:55:45.114730 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.114689 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2"] Sep 10 04:55:45.115392 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.115371 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="087ee327-2005-4005-a929-865122a9d431" containerName="maas-api" Sep 10 04:55:45.115517 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.115395 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="087ee327-2005-4005-a929-865122a9d431" containerName="maas-api" Sep 10 04:55:45.115597 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.115518 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="087ee327-2005-4005-a929-865122a9d431" containerName="maas-api" Sep 10 04:55:45.120156 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.120134 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.122802 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.122779 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d778192d-serving-cert\"" Sep 10 04:55:45.129822 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.129748 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2"] Sep 10 04:55:45.195512 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.195480 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-maas-api-tls\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.195725 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.195520 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e341f108-cf31-4309-a75d-275d03e4862a-openshift-service-ca\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.195725 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.195651 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmm4\" (UniqueName: \"kubernetes.io/projected/e341f108-cf31-4309-a75d-275d03e4862a-kube-api-access-4hmm4\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.195857 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.195770 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-metrics-certs\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.296966 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.296927 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-metrics-certs\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.297183 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.297017 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-maas-api-tls\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.297183 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.297052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e341f108-cf31-4309-a75d-275d03e4862a-openshift-service-ca\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.297183 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.297153 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmm4\" (UniqueName: \"kubernetes.io/projected/e341f108-cf31-4309-a75d-275d03e4862a-kube-api-access-4hmm4\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.297400 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:55:45.297375 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-serving-cert: secret "maas-api-e2e-migrate-d778192d-serving-cert" not found Sep 10 04:55:45.297489 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:55:45.297474 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-maas-api-tls podName:e341f108-cf31-4309-a75d-275d03e4862a nodeName:}" failed. No retries permitted until 2026-09-10 04:55:45.797436571 +0000 UTC m=+865.301269629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-maas-api-tls") pod "maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" (UID: "e341f108-cf31-4309-a75d-275d03e4862a") : secret "maas-api-e2e-migrate-d778192d-serving-cert" not found Sep 10 04:55:45.300595 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.298614 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e341f108-cf31-4309-a75d-275d03e4862a-openshift-service-ca\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.300750 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.300605 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-metrics-certs\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.306641 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.306608 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmm4\" (UniqueName: \"kubernetes.io/projected/e341f108-cf31-4309-a75d-275d03e4862a-kube-api-access-4hmm4\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.803555 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.803514 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-maas-api-tls\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.807911 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.807878 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-maas-api-tls\") pod \"maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:45.964095 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.964052 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz"] Sep 10 04:55:45.968053 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.968030 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:55:45.970787 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.970764 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d778192d-dockercfg-xw46s\"" Sep 10 04:55:45.971110 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.971093 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d778192d\"" Sep 10 04:55:45.976357 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:45.976328 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz"] Sep 10 04:55:46.034488 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.033966 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:46.113251 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.113206 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk55x\" (UniqueName: \"kubernetes.io/projected/d040158b-3fda-481b-a624-5cc7ee0a988e-kube-api-access-xk55x\") pod \"payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz\" (UID: \"d040158b-3fda-481b-a624-5cc7ee0a988e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:55:46.113479 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.113306 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d040158b-3fda-481b-a624-5cc7ee0a988e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz\" (UID: \"d040158b-3fda-481b-a624-5cc7ee0a988e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:55:46.214725 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.214693 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk55x\" (UniqueName: \"kubernetes.io/projected/d040158b-3fda-481b-a624-5cc7ee0a988e-kube-api-access-xk55x\") pod \"payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz\" (UID: \"d040158b-3fda-481b-a624-5cc7ee0a988e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:55:46.215100 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.214762 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d040158b-3fda-481b-a624-5cc7ee0a988e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz\" (UID: \"d040158b-3fda-481b-a624-5cc7ee0a988e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:55:46.215507 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.215489 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d040158b-3fda-481b-a624-5cc7ee0a988e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz\" (UID: \"d040158b-3fda-481b-a624-5cc7ee0a988e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:55:46.223114 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.223082 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk55x\" (UniqueName: \"kubernetes.io/projected/d040158b-3fda-481b-a624-5cc7ee0a988e-kube-api-access-xk55x\") pod \"payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz\" (UID: \"d040158b-3fda-481b-a624-5cc7ee0a988e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:55:46.281803 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.281769 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:55:46.337919 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.337703 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2"] Sep 10 04:55:46.418824 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.418785 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:55:46.525825 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.525782 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-credential-socket\") pod \"9076fab0-fd5c-4195-8f21-89332de59d84\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " Sep 10 04:55:46.525825 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.525834 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4xxf\" (UniqueName: \"kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-kube-api-access-p4xxf\") pod \"9076fab0-fd5c-4195-8f21-89332de59d84\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " Sep 10 04:55:46.526048 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.525896 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9076fab0-fd5c-4195-8f21-89332de59d84-istio-podinfo\") pod \"9076fab0-fd5c-4195-8f21-89332de59d84\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " Sep 10 04:55:46.526048 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.525916 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-data\") pod \"9076fab0-fd5c-4195-8f21-89332de59d84\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " Sep 10 04:55:46.526148 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526100 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-envoy\") pod \"9076fab0-fd5c-4195-8f21-89332de59d84\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " Sep 10 04:55:46.526148 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526120 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9076fab0-fd5c-4195-8f21-89332de59d84" (UID: "9076fab0-fd5c-4195-8f21-89332de59d84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:55:46.526228 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526176 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-socket\") pod \"9076fab0-fd5c-4195-8f21-89332de59d84\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " Sep 10 04:55:46.526270 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526240 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9076fab0-fd5c-4195-8f21-89332de59d84-istiod-ca-cert\") pod \"9076fab0-fd5c-4195-8f21-89332de59d84\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " Sep 10 04:55:46.526270 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526237 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-data" (OuterVolumeSpecName: "istio-data") pod "9076fab0-fd5c-4195-8f21-89332de59d84" (UID: "9076fab0-fd5c-4195-8f21-89332de59d84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:55:46.526270 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526261 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-certs\") pod \"9076fab0-fd5c-4195-8f21-89332de59d84\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " Sep 10 04:55:46.526437 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526295 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-istio-token\") pod \"9076fab0-fd5c-4195-8f21-89332de59d84\" (UID: \"9076fab0-fd5c-4195-8f21-89332de59d84\") " Sep 10 04:55:46.526536 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526432 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9076fab0-fd5c-4195-8f21-89332de59d84" (UID: "9076fab0-fd5c-4195-8f21-89332de59d84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:55:46.526670 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526647 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9076fab0-fd5c-4195-8f21-89332de59d84-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9076fab0-fd5c-4195-8f21-89332de59d84" (UID: "9076fab0-fd5c-4195-8f21-89332de59d84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:55:46.526762 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526704 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9076fab0-fd5c-4195-8f21-89332de59d84" (UID: "9076fab0-fd5c-4195-8f21-89332de59d84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:55:46.526827 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526787 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-data\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:46.526882 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526822 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:46.526882 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526838 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9076fab0-fd5c-4195-8f21-89332de59d84-istiod-ca-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:46.526882 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.526853 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-credential-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:46.528523 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.528491 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9076fab0-fd5c-4195-8f21-89332de59d84-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9076fab0-fd5c-4195-8f21-89332de59d84" (UID: "9076fab0-fd5c-4195-8f21-89332de59d84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 04:55:46.528625 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.528554 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-istio-token" (OuterVolumeSpecName: "istio-token") pod "9076fab0-fd5c-4195-8f21-89332de59d84" (UID: "9076fab0-fd5c-4195-8f21-89332de59d84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:55:46.528672 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.528647 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9076fab0-fd5c-4195-8f21-89332de59d84" (UID: "9076fab0-fd5c-4195-8f21-89332de59d84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:55:46.528672 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.528657 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-kube-api-access-p4xxf" (OuterVolumeSpecName: "kube-api-access-p4xxf") pod "9076fab0-fd5c-4195-8f21-89332de59d84" (UID: "9076fab0-fd5c-4195-8f21-89332de59d84"). InnerVolumeSpecName "kube-api-access-p4xxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:55:46.628387 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.628260 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4xxf\" (UniqueName: \"kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-kube-api-access-p4xxf\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:46.628387 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.628317 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9076fab0-fd5c-4195-8f21-89332de59d84-istio-podinfo\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:46.628387 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.628331 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-istio-envoy\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:46.628387 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.628343 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9076fab0-fd5c-4195-8f21-89332de59d84-workload-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:46.628387 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.628355 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9076fab0-fd5c-4195-8f21-89332de59d84-istio-token\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:55:46.651168 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.651136 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz"] Sep 10 04:55:46.653273 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:55:46.653245 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd040158b_3fda_481b_a624_5cc7ee0a988e.slice/crio-536a095c751b02594c4aad577a63a3d8eca17ee6858a610b37639ecd5d7d110b WatchSource:0}: Error finding container 536a095c751b02594c4aad577a63a3d8eca17ee6858a610b37639ecd5d7d110b: Status 404 returned error can't find the container with id 536a095c751b02594c4aad577a63a3d8eca17ee6858a610b37639ecd5d7d110b Sep 10 04:55:46.731749 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.731713 2572 generic.go:356] "Generic (PLEG): container finished" podID="9076fab0-fd5c-4195-8f21-89332de59d84" containerID="c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781" exitCode=0 Sep 10 04:55:46.731898 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.731771 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" Sep 10 04:55:46.731898 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.731788 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" event={"ID":"9076fab0-fd5c-4195-8f21-89332de59d84","Type":"ContainerDied","Data":"c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781"} Sep 10 04:55:46.731898 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.731849 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t" event={"ID":"9076fab0-fd5c-4195-8f21-89332de59d84","Type":"ContainerDied","Data":"411a0d5818054e180e72ec41a1414167a8fbb2a8e5d4c8fc6834314779efb64e"} Sep 10 04:55:46.731898 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.731880 2572 scope.go:117] "RemoveContainer" containerID="c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781" Sep 10 04:55:46.733905 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.733880 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" event={"ID":"d040158b-3fda-481b-a624-5cc7ee0a988e","Type":"ContainerStarted","Data":"536a095c751b02594c4aad577a63a3d8eca17ee6858a610b37639ecd5d7d110b"} Sep 10 04:55:46.735436 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.735407 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" event={"ID":"e341f108-cf31-4309-a75d-275d03e4862a","Type":"ContainerStarted","Data":"baa1116e8dc4fe0c12f5df177860236001b005db7dc19ce0835298b2bcefc0f8"} Sep 10 04:55:46.741964 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.741940 2572 scope.go:117] "RemoveContainer" containerID="7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b" Sep 10 04:55:46.751725 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.751699 2572 scope.go:117] "RemoveContainer" containerID="c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781" Sep 10 04:55:46.752164 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:55:46.752128 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781\": container with ID starting with c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781 not found: ID does not exist" containerID="c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781" Sep 10 04:55:46.752248 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.752179 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781"} err="failed to get container status \"c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781\": rpc error: code = NotFound desc = could not find container \"c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781\": container with ID starting with c4907f72b9a1d4696c6b5782fd5c0e7418a09a1d08e397393011b9de28a83781 not found: ID does not exist" Sep 10 04:55:46.752248 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.752206 2572 scope.go:117] "RemoveContainer" containerID="7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b" Sep 10 04:55:46.752578 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:55:46.752555 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b\": container with ID starting with 7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b not found: ID does not exist" containerID="7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b" Sep 10 04:55:46.752681 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.752582 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b"} err="failed to get container status \"7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b\": rpc error: code = NotFound desc = could not find container \"7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b\": container with ID starting with 7f2dd06e3f16aa3494839abde9f6dd29da397c23915f248efcde2f780c89251b not found: ID does not exist" Sep 10 04:55:46.762399 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.762363 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t"] Sep 10 04:55:46.777975 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:46.777934 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e79d9e97-openshift-default-6f46b4fcbb-g8l9t"] Sep 10 04:55:47.037932 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:47.037898 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" path="/var/lib/kubelet/pods/9076fab0-fd5c-4195-8f21-89332de59d84/volumes" Sep 10 04:55:47.743261 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:47.743226 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" event={"ID":"d040158b-3fda-481b-a624-5cc7ee0a988e","Type":"ContainerStarted","Data":"6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4"} Sep 10 04:55:47.743766 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:47.743365 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:55:47.744759 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:47.744735 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" event={"ID":"e341f108-cf31-4309-a75d-275d03e4862a","Type":"ContainerStarted","Data":"646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a"} Sep 10 04:55:47.744863 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:47.744807 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:47.761062 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:47.761008 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" podStartSLOduration=2.760992038 podStartE2EDuration="2.760992038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:55:47.759469428 +0000 UTC m=+867.263302481" watchObservedRunningTime="2026-09-10 04:55:47.760992038 +0000 UTC m=+867.264825124" Sep 10 04:55:47.776113 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:47.776061 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" podStartSLOduration=2.303921729 podStartE2EDuration="2.776043494s" totalImagesPullingTime="472.121765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:55:47.775172296 +0000 UTC m=+867.279005358" watchObservedRunningTime="2026-09-10 04:55:47.776043494 +0000 UTC m=+867.279876555" Sep 10 04:55:53.756004 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:53.755971 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:55:58.753007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:55:58.752975 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:56:16.323162 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.323084 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2"] Sep 10 04:56:16.323630 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.323316 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" podUID="e341f108-cf31-4309-a75d-275d03e4862a" containerName="maas-api" containerID="cri-o://646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a" gracePeriod=30 Sep 10 04:56:16.395515 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.395480 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz"] Sep 10 04:56:16.395804 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.395754 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" podUID="d040158b-3fda-481b-a624-5cc7ee0a988e" containerName="payload-pre-processing" containerID="cri-o://6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4" gracePeriod=30 Sep 10 04:56:16.637697 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.637663 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:56:16.667899 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.667874 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:56:16.740311 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.740278 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e341f108-cf31-4309-a75d-275d03e4862a-openshift-service-ca\") pod \"e341f108-cf31-4309-a75d-275d03e4862a\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " Sep 10 04:56:16.740544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.740335 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-maas-api-tls\") pod \"e341f108-cf31-4309-a75d-275d03e4862a\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " Sep 10 04:56:16.740544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.740366 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hmm4\" (UniqueName: \"kubernetes.io/projected/e341f108-cf31-4309-a75d-275d03e4862a-kube-api-access-4hmm4\") pod \"e341f108-cf31-4309-a75d-275d03e4862a\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " Sep 10 04:56:16.740544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.740529 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk55x\" (UniqueName: \"kubernetes.io/projected/d040158b-3fda-481b-a624-5cc7ee0a988e-kube-api-access-xk55x\") pod \"d040158b-3fda-481b-a624-5cc7ee0a988e\" (UID: \"d040158b-3fda-481b-a624-5cc7ee0a988e\") " Sep 10 04:56:16.740725 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.740579 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-metrics-certs\") pod \"e341f108-cf31-4309-a75d-275d03e4862a\" (UID: \"e341f108-cf31-4309-a75d-275d03e4862a\") " Sep 10 04:56:16.740725 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.740673 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d040158b-3fda-481b-a624-5cc7ee0a988e-plugins-config-volume\") pod \"d040158b-3fda-481b-a624-5cc7ee0a988e\" (UID: \"d040158b-3fda-481b-a624-5cc7ee0a988e\") " Sep 10 04:56:16.741174 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.741142 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d040158b-3fda-481b-a624-5cc7ee0a988e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d040158b-3fda-481b-a624-5cc7ee0a988e" (UID: "d040158b-3fda-481b-a624-5cc7ee0a988e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:56:16.741505 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.741480 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e341f108-cf31-4309-a75d-275d03e4862a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e341f108-cf31-4309-a75d-275d03e4862a" (UID: "e341f108-cf31-4309-a75d-275d03e4862a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:56:16.743108 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.743069 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e341f108-cf31-4309-a75d-275d03e4862a-kube-api-access-4hmm4" (OuterVolumeSpecName: "kube-api-access-4hmm4") pod "e341f108-cf31-4309-a75d-275d03e4862a" (UID: "e341f108-cf31-4309-a75d-275d03e4862a"). InnerVolumeSpecName "kube-api-access-4hmm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:56:16.743251 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.743173 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d040158b-3fda-481b-a624-5cc7ee0a988e-kube-api-access-xk55x" (OuterVolumeSpecName: "kube-api-access-xk55x") pod "d040158b-3fda-481b-a624-5cc7ee0a988e" (UID: "d040158b-3fda-481b-a624-5cc7ee0a988e"). InnerVolumeSpecName "kube-api-access-xk55x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:56:16.743665 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.743644 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e341f108-cf31-4309-a75d-275d03e4862a" (UID: "e341f108-cf31-4309-a75d-275d03e4862a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:56:16.743752 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.743690 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e341f108-cf31-4309-a75d-275d03e4862a" (UID: "e341f108-cf31-4309-a75d-275d03e4862a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:56:16.843098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.842498 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-maas-api-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:16.843098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.842557 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hmm4\" (UniqueName: \"kubernetes.io/projected/e341f108-cf31-4309-a75d-275d03e4862a-kube-api-access-4hmm4\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:16.843098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.842583 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk55x\" (UniqueName: \"kubernetes.io/projected/d040158b-3fda-481b-a624-5cc7ee0a988e-kube-api-access-xk55x\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:16.843098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.842603 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e341f108-cf31-4309-a75d-275d03e4862a-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:16.843098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.842629 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d040158b-3fda-481b-a624-5cc7ee0a988e-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:16.843098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.842651 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e341f108-cf31-4309-a75d-275d03e4862a-openshift-service-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:16.910090 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.910002 2572 generic.go:356] "Generic (PLEG): container finished" podID="d040158b-3fda-481b-a624-5cc7ee0a988e" containerID="6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4" exitCode=0 Sep 10 04:56:16.910090 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.910050 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" Sep 10 04:56:16.910342 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.910085 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" event={"ID":"d040158b-3fda-481b-a624-5cc7ee0a988e","Type":"ContainerDied","Data":"6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4"} Sep 10 04:56:16.910342 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.910149 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz" event={"ID":"d040158b-3fda-481b-a624-5cc7ee0a988e","Type":"ContainerDied","Data":"536a095c751b02594c4aad577a63a3d8eca17ee6858a610b37639ecd5d7d110b"} Sep 10 04:56:16.910342 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.910178 2572 scope.go:117] "RemoveContainer" containerID="6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4" Sep 10 04:56:16.911636 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.911616 2572 generic.go:356] "Generic (PLEG): container finished" podID="e341f108-cf31-4309-a75d-275d03e4862a" containerID="646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a" exitCode=0 Sep 10 04:56:16.911746 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.911651 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" Sep 10 04:56:16.911746 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.911698 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" event={"ID":"e341f108-cf31-4309-a75d-275d03e4862a","Type":"ContainerDied","Data":"646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a"} Sep 10 04:56:16.911843 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.911755 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2" event={"ID":"e341f108-cf31-4309-a75d-275d03e4862a","Type":"ContainerDied","Data":"baa1116e8dc4fe0c12f5df177860236001b005db7dc19ce0835298b2bcefc0f8"} Sep 10 04:56:16.921650 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.921632 2572 scope.go:117] "RemoveContainer" containerID="6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4" Sep 10 04:56:16.921961 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:56:16.921941 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4\": container with ID starting with 6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4 not found: ID does not exist" containerID="6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4" Sep 10 04:56:16.922097 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.921973 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4"} err="failed to get container status \"6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4\": rpc error: code = NotFound desc = could not find container \"6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4\": container with ID starting with 6b80c7c185edd2289bf286f35050614ebe3c1bdac5e3dbdc18bc66b3f67a48a4 not found: ID does not exist" Sep 10 04:56:16.922097 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.921992 2572 scope.go:117] "RemoveContainer" containerID="646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a" Sep 10 04:56:16.930918 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.930775 2572 scope.go:117] "RemoveContainer" containerID="646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a" Sep 10 04:56:16.931265 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:56:16.931243 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a\": container with ID starting with 646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a not found: ID does not exist" containerID="646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a" Sep 10 04:56:16.931330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.931273 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a"} err="failed to get container status \"646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a\": rpc error: code = NotFound desc = could not find container \"646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a\": container with ID starting with 646389d78daeec3279df1c64cd125a696172cf544d3b284231209caa4c808d6a not found: ID does not exist" Sep 10 04:56:16.934535 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.934512 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz"] Sep 10 04:56:16.936554 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.936532 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d778192d-887fc4b94-fkrmz"] Sep 10 04:56:16.945409 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.945385 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2"] Sep 10 04:56:16.949172 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:16.949143 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d778192d-7684b7677d-2fqh2"] Sep 10 04:56:17.038494 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.038431 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d040158b-3fda-481b-a624-5cc7ee0a988e" path="/var/lib/kubelet/pods/d040158b-3fda-481b-a624-5cc7ee0a988e/volumes" Sep 10 04:56:17.038953 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.038934 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e341f108-cf31-4309-a75d-275d03e4862a" path="/var/lib/kubelet/pods/e341f108-cf31-4309-a75d-275d03e4862a/volumes" Sep 10 04:56:17.543666 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.543630 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl"] Sep 10 04:56:17.544209 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.544179 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" Sep 10 04:56:17.544209 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.544202 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" Sep 10 04:56:17.544415 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.544221 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" Sep 10 04:56:17.544415 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.544230 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" Sep 10 04:56:17.544415 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.544240 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d040158b-3fda-481b-a624-5cc7ee0a988e" containerName="payload-pre-processing" Sep 10 04:56:17.544415 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.544246 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d040158b-3fda-481b-a624-5cc7ee0a988e" containerName="payload-pre-processing" Sep 10 04:56:17.544415 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.544252 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e341f108-cf31-4309-a75d-275d03e4862a" containerName="maas-api" Sep 10 04:56:17.544415 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.544257 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e341f108-cf31-4309-a75d-275d03e4862a" containerName="maas-api" Sep 10 04:56:17.544415 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.544347 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" Sep 10 04:56:17.544415 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.544355 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d040158b-3fda-481b-a624-5cc7ee0a988e" containerName="payload-pre-processing" Sep 10 04:56:17.544415 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.544362 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e341f108-cf31-4309-a75d-275d03e4862a" containerName="maas-api" Sep 10 04:56:17.547908 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.547886 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.550544 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.550522 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-eb4df850-openshift-default-dockercfg-d4lq6\"" Sep 10 04:56:17.560972 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.560934 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl"] Sep 10 04:56:17.652216 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.652108 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-envoy\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.652388 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.652222 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-data\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.652388 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.652261 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-token\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.652388 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.652323 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-socket\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.652612 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.652408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-credential-socket\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.652612 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.652474 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxnd8\" (UniqueName: \"kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-kube-api-access-xxnd8\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.652612 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.652507 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-podinfo\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.652612 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.652589 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-certs\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.652742 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.652629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istiod-ca-cert\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.753855 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.753810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-podinfo\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754061 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.753888 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-certs\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754061 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.753917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istiod-ca-cert\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754204 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.754065 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-envoy\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754204 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.754174 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-data\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754312 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.754220 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-token\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754312 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.754291 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-socket\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754418 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.754338 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-credential-socket\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754418 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.754366 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxnd8\" (UniqueName: \"kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-kube-api-access-xxnd8\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754648 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.754622 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-data\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754753 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.754638 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-certs\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754823 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.754787 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istiod-ca-cert\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754870 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.754839 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-socket\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.754910 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.754888 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-credential-socket\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.756438 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.756413 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-envoy\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.756693 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.756674 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-podinfo\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.762133 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.762096 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-token\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.762253 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.762189 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxnd8\" (UniqueName: \"kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-kube-api-access-xxnd8\") pod \"e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:17.861977 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:17.861886 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:18.140134 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:18.140107 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl"] Sep 10 04:56:18.142097 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:56:18.142065 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747e6ce6_9b54_45f9_b419_435f9b07ce7b.slice/crio-200e4789aad39da4657f4b60902b80fcbe65d5ed58f74633a7388ba09dcfae8c WatchSource:0}: Error finding container 200e4789aad39da4657f4b60902b80fcbe65d5ed58f74633a7388ba09dcfae8c: Status 404 returned error can't find the container with id 200e4789aad39da4657f4b60902b80fcbe65d5ed58f74633a7388ba09dcfae8c Sep 10 04:56:18.147397 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:18.147364 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:56:18.147505 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:18.147476 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:56:18.147553 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:18.147509 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:56:18.930611 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:18.930574 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" event={"ID":"747e6ce6-9b54-45f9-b419-435f9b07ce7b","Type":"ContainerStarted","Data":"e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd"} Sep 10 04:56:18.931075 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:18.930653 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" event={"ID":"747e6ce6-9b54-45f9-b419-435f9b07ce7b","Type":"ContainerStarted","Data":"200e4789aad39da4657f4b60902b80fcbe65d5ed58f74633a7388ba09dcfae8c"} Sep 10 04:56:18.951652 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:18.951593 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" podStartSLOduration=1.9515751300000002 podStartE2EDuration="1.95157513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:56:18.949300257 +0000 UTC m=+898.453133334" watchObservedRunningTime="2026-09-10 04:56:18.95157513 +0000 UTC m=+898.455408192" Sep 10 04:56:19.862498 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:19.862432 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:19.867627 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:19.867597 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:19.935875 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:19.935844 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:19.936747 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:19.936727 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:22.077598 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.077561 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq"] Sep 10 04:56:22.078079 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.078065 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9076fab0-fd5c-4195-8f21-89332de59d84" containerName="istio-proxy" Sep 10 04:56:22.082684 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.082662 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:22.085328 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.085301 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-eb4df850\"" Sep 10 04:56:22.093098 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.093068 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq"] Sep 10 04:56:22.095336 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.095309 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w"] Sep 10 04:56:22.098985 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.098941 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:22.101415 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.101389 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-eb4df850-dockercfg-jk9p5\"" Sep 10 04:56:22.107763 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.107619 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w"] Sep 10 04:56:22.212245 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.210007 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mfkx\" (UniqueName: \"kubernetes.io/projected/d1bde348-7a53-45c9-99c3-02ff3022b6f7-kube-api-access-9mfkx\") pod \"payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w\" (UID: \"d1bde348-7a53-45c9-99c3-02ff3022b6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:22.212245 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.210068 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bde348-7a53-45c9-99c3-02ff3022b6f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w\" (UID: \"d1bde348-7a53-45c9-99c3-02ff3022b6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:22.212245 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.210136 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a788df-4cb8-47cb-9fce-58248012baa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq\" (UID: \"89a788df-4cb8-47cb-9fce-58248012baa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:22.212245 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.210180 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2g68\" (UniqueName: \"kubernetes.io/projected/89a788df-4cb8-47cb-9fce-58248012baa5-kube-api-access-s2g68\") pod \"payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq\" (UID: \"89a788df-4cb8-47cb-9fce-58248012baa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:22.311425 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.311385 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mfkx\" (UniqueName: \"kubernetes.io/projected/d1bde348-7a53-45c9-99c3-02ff3022b6f7-kube-api-access-9mfkx\") pod \"payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w\" (UID: \"d1bde348-7a53-45c9-99c3-02ff3022b6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:22.311633 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.311439 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bde348-7a53-45c9-99c3-02ff3022b6f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w\" (UID: \"d1bde348-7a53-45c9-99c3-02ff3022b6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:22.311633 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.311516 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a788df-4cb8-47cb-9fce-58248012baa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq\" (UID: \"89a788df-4cb8-47cb-9fce-58248012baa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:22.311633 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.311554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2g68\" (UniqueName: \"kubernetes.io/projected/89a788df-4cb8-47cb-9fce-58248012baa5-kube-api-access-s2g68\") pod \"payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq\" (UID: \"89a788df-4cb8-47cb-9fce-58248012baa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:22.312216 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.312194 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a788df-4cb8-47cb-9fce-58248012baa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq\" (UID: \"89a788df-4cb8-47cb-9fce-58248012baa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:22.312406 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.312388 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bde348-7a53-45c9-99c3-02ff3022b6f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w\" (UID: \"d1bde348-7a53-45c9-99c3-02ff3022b6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:22.319463 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.319423 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mfkx\" (UniqueName: \"kubernetes.io/projected/d1bde348-7a53-45c9-99c3-02ff3022b6f7-kube-api-access-9mfkx\") pod \"payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w\" (UID: \"d1bde348-7a53-45c9-99c3-02ff3022b6f7\") " pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:22.319566 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.319547 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2g68\" (UniqueName: \"kubernetes.io/projected/89a788df-4cb8-47cb-9fce-58248012baa5-kube-api-access-s2g68\") pod \"payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq\" (UID: \"89a788df-4cb8-47cb-9fce-58248012baa5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:22.394422 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.394332 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:22.412802 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.412771 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:22.670432 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.670400 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq"] Sep 10 04:56:22.672052 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:56:22.672024 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a788df_4cb8_47cb_9fce_58248012baa5.slice/crio-f8e7783e98ceeeed871ed7ccec9c23901da2a35884be67be3cd59c830adf9e10 WatchSource:0}: Error finding container f8e7783e98ceeeed871ed7ccec9c23901da2a35884be67be3cd59c830adf9e10: Status 404 returned error can't find the container with id f8e7783e98ceeeed871ed7ccec9c23901da2a35884be67be3cd59c830adf9e10 Sep 10 04:56:22.697438 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.697408 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w"] Sep 10 04:56:22.708609 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:56:22.708578 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bde348_7a53_45c9_99c3_02ff3022b6f7.slice/crio-ae1139a6b21ca2d9195ab64512e5654fef57cab615171ea65c743f828b9d181a WatchSource:0}: Error finding container ae1139a6b21ca2d9195ab64512e5654fef57cab615171ea65c743f828b9d181a: Status 404 returned error can't find the container with id ae1139a6b21ca2d9195ab64512e5654fef57cab615171ea65c743f828b9d181a Sep 10 04:56:22.957027 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.956912 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" event={"ID":"d1bde348-7a53-45c9-99c3-02ff3022b6f7","Type":"ContainerStarted","Data":"21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6"} Sep 10 04:56:22.957027 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.956986 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" event={"ID":"d1bde348-7a53-45c9-99c3-02ff3022b6f7","Type":"ContainerStarted","Data":"ae1139a6b21ca2d9195ab64512e5654fef57cab615171ea65c743f828b9d181a"} Sep 10 04:56:22.957027 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.957014 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:22.958490 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.958442 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" event={"ID":"89a788df-4cb8-47cb-9fce-58248012baa5","Type":"ContainerStarted","Data":"8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c"} Sep 10 04:56:22.958648 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.958628 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:22.958744 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.958653 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" event={"ID":"89a788df-4cb8-47cb-9fce-58248012baa5","Type":"ContainerStarted","Data":"f8e7783e98ceeeed871ed7ccec9c23901da2a35884be67be3cd59c830adf9e10"} Sep 10 04:56:22.974749 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.974691 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" podStartSLOduration=0.974669076 podStartE2EDuration="974.669076ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:56:22.973030976 +0000 UTC m=+902.476864031" watchObservedRunningTime="2026-09-10 04:56:22.974669076 +0000 UTC m=+902.478502138" Sep 10 04:56:22.989769 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:22.989711 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" podStartSLOduration=0.989692014 podStartE2EDuration="989.692014ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:56:22.987609896 +0000 UTC m=+902.491442960" watchObservedRunningTime="2026-09-10 04:56:22.989692014 +0000 UTC m=+902.493525078" Sep 10 04:56:33.965683 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:33.965646 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:33.966293 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:33.966081 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:43.072086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:43.071992 2572 generic.go:356] "Generic (PLEG): container finished" podID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerID="e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd" exitCode=0 Sep 10 04:56:43.072086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:43.072066 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" event={"ID":"747e6ce6-9b54-45f9-b419-435f9b07ce7b","Type":"ContainerDied","Data":"e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd"} Sep 10 04:56:43.072556 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:43.072492 2572 scope.go:117] "RemoveContainer" containerID="e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd" Sep 10 04:56:43.073712 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:43.073681 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:56:43.073803 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:43.073746 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:56:43.073803 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:43.073778 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:56:44.080441 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:44.080405 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" event={"ID":"747e6ce6-9b54-45f9-b419-435f9b07ce7b","Type":"ContainerStarted","Data":"66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300"} Sep 10 04:56:44.862478 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:44.862417 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:44.862688 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:44.862666 2572 patch_prober.go:28] interesting pod/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 04:56:44.862738 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:44.862717 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 04:56:45.862948 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:45.862905 2572 patch_prober.go:28] interesting pod/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 04:56:45.863344 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:45.862970 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 04:56:46.862834 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:46.862795 2572 patch_prober.go:28] interesting pod/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 04:56:46.863041 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:46.862857 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 04:56:47.862857 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:47.862819 2572 patch_prober.go:28] interesting pod/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 04:56:47.863050 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:47.862881 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 04:56:48.862478 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:48.862415 2572 patch_prober.go:28] interesting pod/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 04:56:48.862666 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:48.862514 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 04:56:49.862729 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:49.862690 2572 patch_prober.go:28] interesting pod/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Sep 10 04:56:49.863199 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:49.862757 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Sep 10 04:56:49.936306 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:49.936259 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:50.866599 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:50.866560 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:50.867586 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:50.867565 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:52.139462 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.139405 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w"] Sep 10 04:56:52.141538 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.140224 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" podUID="d1bde348-7a53-45c9-99c3-02ff3022b6f7" containerName="payload-processing" containerID="cri-o://21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6" gracePeriod=30 Sep 10 04:56:52.141942 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.141099 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq"] Sep 10 04:56:52.142286 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.142259 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" podUID="89a788df-4cb8-47cb-9fce-58248012baa5" containerName="payload-pre-processing" containerID="cri-o://8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c" gracePeriod=30 Sep 10 04:56:52.434177 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.434147 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:52.481916 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.481883 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:52.539173 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.539134 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mfkx\" (UniqueName: \"kubernetes.io/projected/d1bde348-7a53-45c9-99c3-02ff3022b6f7-kube-api-access-9mfkx\") pod \"d1bde348-7a53-45c9-99c3-02ff3022b6f7\" (UID: \"d1bde348-7a53-45c9-99c3-02ff3022b6f7\") " Sep 10 04:56:52.539377 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.539241 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bde348-7a53-45c9-99c3-02ff3022b6f7-plugins-config-volume\") pod \"d1bde348-7a53-45c9-99c3-02ff3022b6f7\" (UID: \"d1bde348-7a53-45c9-99c3-02ff3022b6f7\") " Sep 10 04:56:52.539657 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.539627 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1bde348-7a53-45c9-99c3-02ff3022b6f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1bde348-7a53-45c9-99c3-02ff3022b6f7" (UID: "d1bde348-7a53-45c9-99c3-02ff3022b6f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:56:52.541325 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.541301 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1bde348-7a53-45c9-99c3-02ff3022b6f7-kube-api-access-9mfkx" (OuterVolumeSpecName: "kube-api-access-9mfkx") pod "d1bde348-7a53-45c9-99c3-02ff3022b6f7" (UID: "d1bde348-7a53-45c9-99c3-02ff3022b6f7"). InnerVolumeSpecName "kube-api-access-9mfkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:56:52.640767 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.640728 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a788df-4cb8-47cb-9fce-58248012baa5-plugins-config-volume\") pod \"89a788df-4cb8-47cb-9fce-58248012baa5\" (UID: \"89a788df-4cb8-47cb-9fce-58248012baa5\") " Sep 10 04:56:52.640949 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.640820 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2g68\" (UniqueName: \"kubernetes.io/projected/89a788df-4cb8-47cb-9fce-58248012baa5-kube-api-access-s2g68\") pod \"89a788df-4cb8-47cb-9fce-58248012baa5\" (UID: \"89a788df-4cb8-47cb-9fce-58248012baa5\") " Sep 10 04:56:52.641129 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.641099 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a788df-4cb8-47cb-9fce-58248012baa5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89a788df-4cb8-47cb-9fce-58248012baa5" (UID: "89a788df-4cb8-47cb-9fce-58248012baa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:56:52.641207 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.641193 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mfkx\" (UniqueName: \"kubernetes.io/projected/d1bde348-7a53-45c9-99c3-02ff3022b6f7-kube-api-access-9mfkx\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:52.641246 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.641211 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a788df-4cb8-47cb-9fce-58248012baa5-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:52.641246 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.641221 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1bde348-7a53-45c9-99c3-02ff3022b6f7-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:52.642970 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.642950 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a788df-4cb8-47cb-9fce-58248012baa5-kube-api-access-s2g68" (OuterVolumeSpecName: "kube-api-access-s2g68") pod "89a788df-4cb8-47cb-9fce-58248012baa5" (UID: "89a788df-4cb8-47cb-9fce-58248012baa5"). InnerVolumeSpecName "kube-api-access-s2g68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:56:52.742422 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:52.742388 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2g68\" (UniqueName: \"kubernetes.io/projected/89a788df-4cb8-47cb-9fce-58248012baa5-kube-api-access-s2g68\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:53.155785 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.155749 2572 generic.go:356] "Generic (PLEG): container finished" podID="d1bde348-7a53-45c9-99c3-02ff3022b6f7" containerID="21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6" exitCode=0 Sep 10 04:56:53.156261 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.155791 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" Sep 10 04:56:53.156261 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.155832 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" event={"ID":"d1bde348-7a53-45c9-99c3-02ff3022b6f7","Type":"ContainerDied","Data":"21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6"} Sep 10 04:56:53.156261 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.155879 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w" event={"ID":"d1bde348-7a53-45c9-99c3-02ff3022b6f7","Type":"ContainerDied","Data":"ae1139a6b21ca2d9195ab64512e5654fef57cab615171ea65c743f828b9d181a"} Sep 10 04:56:53.156261 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.155910 2572 scope.go:117] "RemoveContainer" containerID="21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6" Sep 10 04:56:53.157344 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.157214 2572 generic.go:356] "Generic (PLEG): container finished" podID="89a788df-4cb8-47cb-9fce-58248012baa5" containerID="8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c" exitCode=0 Sep 10 04:56:53.157344 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.157259 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" event={"ID":"89a788df-4cb8-47cb-9fce-58248012baa5","Type":"ContainerDied","Data":"8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c"} Sep 10 04:56:53.157344 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.157275 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" Sep 10 04:56:53.157344 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.157304 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq" event={"ID":"89a788df-4cb8-47cb-9fce-58248012baa5","Type":"ContainerDied","Data":"f8e7783e98ceeeed871ed7ccec9c23901da2a35884be67be3cd59c830adf9e10"} Sep 10 04:56:53.165394 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.165373 2572 scope.go:117] "RemoveContainer" containerID="21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6" Sep 10 04:56:53.165791 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:56:53.165768 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6\": container with ID starting with 21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6 not found: ID does not exist" containerID="21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6" Sep 10 04:56:53.165883 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.165803 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6"} err="failed to get container status \"21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6\": rpc error: code = NotFound desc = could not find container \"21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6\": container with ID starting with 21e109716dc77d4cbeb22d6a2d74e0c2b03da25938f4623014a2edb1ebb27bc6 not found: ID does not exist" Sep 10 04:56:53.165883 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.165826 2572 scope.go:117] "RemoveContainer" containerID="8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c" Sep 10 04:56:53.174844 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.174822 2572 scope.go:117] "RemoveContainer" containerID="8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c" Sep 10 04:56:53.175185 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:56:53.175165 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c\": container with ID starting with 8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c not found: ID does not exist" containerID="8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c" Sep 10 04:56:53.175234 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.175194 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c"} err="failed to get container status \"8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c\": rpc error: code = NotFound desc = could not find container \"8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c\": container with ID starting with 8777c3e5de88cb25a038a1a83753be4657b86bfb9eb176aee03dc83bbf2bf32c not found: ID does not exist" Sep 10 04:56:53.176075 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.176056 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq"] Sep 10 04:56:53.180027 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.180002 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb4df850-54578f59c4-j4jpq"] Sep 10 04:56:53.190812 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.190775 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w"] Sep 10 04:56:53.194963 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.194937 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb4df850-749bb878d7-dc24w"] Sep 10 04:56:53.454086 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.453985 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl"] Sep 10 04:56:53.454265 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:53.454231 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" containerID="cri-o://66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300" gracePeriod=30 Sep 10 04:56:55.037052 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:55.037008 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89a788df-4cb8-47cb-9fce-58248012baa5" path="/var/lib/kubelet/pods/89a788df-4cb8-47cb-9fce-58248012baa5/volumes" Sep 10 04:56:55.037622 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:55.037601 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1bde348-7a53-45c9-99c3-02ff3022b6f7" path="/var/lib/kubelet/pods/d1bde348-7a53-45c9-99c3-02ff3022b6f7/volumes" Sep 10 04:56:58.723329 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.723301 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:58.804183 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804149 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-envoy\") pod \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " Sep 10 04:56:58.804360 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804209 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-credential-socket\") pod \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " Sep 10 04:56:58.804360 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804231 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-certs\") pod \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " Sep 10 04:56:58.804360 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804247 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istiod-ca-cert\") pod \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " Sep 10 04:56:58.804360 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804277 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-token\") pod \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " Sep 10 04:56:58.804360 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804296 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-data\") pod \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " Sep 10 04:56:58.804360 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804334 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxnd8\" (UniqueName: \"kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-kube-api-access-xxnd8\") pod \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " Sep 10 04:56:58.804697 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804387 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-podinfo\") pod \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " Sep 10 04:56:58.804697 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804427 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-socket\") pod \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\" (UID: \"747e6ce6-9b54-45f9-b419-435f9b07ce7b\") " Sep 10 04:56:58.804697 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804620 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "747e6ce6-9b54-45f9-b419-435f9b07ce7b" (UID: "747e6ce6-9b54-45f9-b419-435f9b07ce7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:56:58.804697 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804652 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "747e6ce6-9b54-45f9-b419-435f9b07ce7b" (UID: "747e6ce6-9b54-45f9-b419-435f9b07ce7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:56:58.804899 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804830 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-credential-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:58.804899 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804852 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:58.804899 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804778 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "747e6ce6-9b54-45f9-b419-435f9b07ce7b" (UID: "747e6ce6-9b54-45f9-b419-435f9b07ce7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:56:58.804899 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804884 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-data" (OuterVolumeSpecName: "istio-data") pod "747e6ce6-9b54-45f9-b419-435f9b07ce7b" (UID: "747e6ce6-9b54-45f9-b419-435f9b07ce7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:56:58.805074 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.804863 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "747e6ce6-9b54-45f9-b419-435f9b07ce7b" (UID: "747e6ce6-9b54-45f9-b419-435f9b07ce7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:56:58.806347 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.806326 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "747e6ce6-9b54-45f9-b419-435f9b07ce7b" (UID: "747e6ce6-9b54-45f9-b419-435f9b07ce7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 04:56:58.806656 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.806639 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-kube-api-access-xxnd8" (OuterVolumeSpecName: "kube-api-access-xxnd8") pod "747e6ce6-9b54-45f9-b419-435f9b07ce7b" (UID: "747e6ce6-9b54-45f9-b419-435f9b07ce7b"). InnerVolumeSpecName "kube-api-access-xxnd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:56:58.807063 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.807041 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-token" (OuterVolumeSpecName: "istio-token") pod "747e6ce6-9b54-45f9-b419-435f9b07ce7b" (UID: "747e6ce6-9b54-45f9-b419-435f9b07ce7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:56:58.807147 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.807043 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "747e6ce6-9b54-45f9-b419-435f9b07ce7b" (UID: "747e6ce6-9b54-45f9-b419-435f9b07ce7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 04:56:58.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.905317 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istiod-ca-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:58.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.905346 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-token\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:58.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.905354 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-data\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:58.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.905362 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxnd8\" (UniqueName: \"kubernetes.io/projected/747e6ce6-9b54-45f9-b419-435f9b07ce7b-kube-api-access-xxnd8\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:58.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.905371 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-podinfo\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:58.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.905379 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-workload-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:58.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:58.905388 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747e6ce6-9b54-45f9-b419-435f9b07ce7b-istio-envoy\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:56:59.075608 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.075572 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5"] Sep 10 04:56:59.076119 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.076104 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" Sep 10 04:56:59.076185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.076121 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" Sep 10 04:56:59.076185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.076135 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a788df-4cb8-47cb-9fce-58248012baa5" containerName="payload-pre-processing" Sep 10 04:56:59.076185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.076141 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a788df-4cb8-47cb-9fce-58248012baa5" containerName="payload-pre-processing" Sep 10 04:56:59.076185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.076154 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1bde348-7a53-45c9-99c3-02ff3022b6f7" containerName="payload-processing" Sep 10 04:56:59.076185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.076159 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1bde348-7a53-45c9-99c3-02ff3022b6f7" containerName="payload-processing" Sep 10 04:56:59.076335 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.076251 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1bde348-7a53-45c9-99c3-02ff3022b6f7" containerName="payload-processing" Sep 10 04:56:59.076335 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.076261 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" Sep 10 04:56:59.076335 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.076269 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="89a788df-4cb8-47cb-9fce-58248012baa5" containerName="payload-pre-processing" Sep 10 04:56:59.079765 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.079741 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:56:59.082277 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.082255 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ead8c2c2\"" Sep 10 04:56:59.082414 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.082282 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ead8c2c2-dockercfg-wnq5r\"" Sep 10 04:56:59.090764 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.090735 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5"] Sep 10 04:56:59.105914 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.105880 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz"] Sep 10 04:56:59.106384 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.106353 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" Sep 10 04:56:59.106384 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.106370 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" Sep 10 04:56:59.106619 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.106460 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerName="istio-proxy" Sep 10 04:56:59.107205 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.107181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62877871-b140-4e14-9aa5-5fd741a310a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5\" (UID: \"62877871-b140-4e14-9aa5-5fd741a310a2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:56:59.107306 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.107248 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kpl8\" (UniqueName: \"kubernetes.io/projected/62877871-b140-4e14-9aa5-5fd741a310a2-kube-api-access-8kpl8\") pod \"payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5\" (UID: \"62877871-b140-4e14-9aa5-5fd741a310a2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:56:59.109718 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.109698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:56:59.125947 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.125911 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz"] Sep 10 04:56:59.200545 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.200442 2572 generic.go:356] "Generic (PLEG): container finished" podID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" containerID="66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300" exitCode=0 Sep 10 04:56:59.200686 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.200646 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" event={"ID":"747e6ce6-9b54-45f9-b419-435f9b07ce7b","Type":"ContainerDied","Data":"66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300"} Sep 10 04:56:59.200740 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.200699 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" event={"ID":"747e6ce6-9b54-45f9-b419-435f9b07ce7b","Type":"ContainerDied","Data":"200e4789aad39da4657f4b60902b80fcbe65d5ed58f74633a7388ba09dcfae8c"} Sep 10 04:56:59.200740 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.200729 2572 scope.go:117] "RemoveContainer" containerID="66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300" Sep 10 04:56:59.200894 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.200880 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl" Sep 10 04:56:59.208069 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.208032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daf4f2e-b620-424b-acb8-196b867647c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz\" (UID: \"2daf4f2e-b620-424b-acb8-196b867647c9\") " pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:56:59.208252 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.208221 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62877871-b140-4e14-9aa5-5fd741a310a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5\" (UID: \"62877871-b140-4e14-9aa5-5fd741a310a2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:56:59.208336 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.208259 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn4rb\" (UniqueName: \"kubernetes.io/projected/2daf4f2e-b620-424b-acb8-196b867647c9-kube-api-access-gn4rb\") pod \"payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz\" (UID: \"2daf4f2e-b620-424b-acb8-196b867647c9\") " pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:56:59.208395 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.208343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kpl8\" (UniqueName: \"kubernetes.io/projected/62877871-b140-4e14-9aa5-5fd741a310a2-kube-api-access-8kpl8\") pod \"payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5\" (UID: \"62877871-b140-4e14-9aa5-5fd741a310a2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:56:59.209619 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.209574 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62877871-b140-4e14-9aa5-5fd741a310a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5\" (UID: \"62877871-b140-4e14-9aa5-5fd741a310a2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:56:59.218524 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.218428 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kpl8\" (UniqueName: \"kubernetes.io/projected/62877871-b140-4e14-9aa5-5fd741a310a2-kube-api-access-8kpl8\") pod \"payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5\" (UID: \"62877871-b140-4e14-9aa5-5fd741a310a2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:56:59.221333 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.221307 2572 scope.go:117] "RemoveContainer" containerID="e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd" Sep 10 04:56:59.223082 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.223058 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl"] Sep 10 04:56:59.227005 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.226982 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-eb4df850-openshift-default-7bf98856b8-2wwvl"] Sep 10 04:56:59.230963 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.230944 2572 scope.go:117] "RemoveContainer" containerID="66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300" Sep 10 04:56:59.231280 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:56:59.231255 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300\": container with ID starting with 66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300 not found: ID does not exist" containerID="66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300" Sep 10 04:56:59.231346 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.231293 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300"} err="failed to get container status \"66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300\": rpc error: code = NotFound desc = could not find container \"66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300\": container with ID starting with 66b7cdfa2166dccfc3f59de7946390c3f7aff8b38e687f6257dfc22e0cff6300 not found: ID does not exist" Sep 10 04:56:59.231346 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.231317 2572 scope.go:117] "RemoveContainer" containerID="e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd" Sep 10 04:56:59.231644 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:56:59.231626 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd\": container with ID starting with e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd not found: ID does not exist" containerID="e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd" Sep 10 04:56:59.231699 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.231650 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd"} err="failed to get container status \"e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd\": rpc error: code = NotFound desc = could not find container \"e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd\": container with ID starting with e40490aa033cf9bd388dc07e1e663196a70cb0e1e2d34265114b4b3af2246ebd not found: ID does not exist" Sep 10 04:56:59.309584 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.309545 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daf4f2e-b620-424b-acb8-196b867647c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz\" (UID: \"2daf4f2e-b620-424b-acb8-196b867647c9\") " pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:56:59.309767 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.309599 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn4rb\" (UniqueName: \"kubernetes.io/projected/2daf4f2e-b620-424b-acb8-196b867647c9-kube-api-access-gn4rb\") pod \"payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz\" (UID: \"2daf4f2e-b620-424b-acb8-196b867647c9\") " pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:56:59.310257 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.310237 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daf4f2e-b620-424b-acb8-196b867647c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz\" (UID: \"2daf4f2e-b620-424b-acb8-196b867647c9\") " pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:56:59.317048 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.317027 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn4rb\" (UniqueName: \"kubernetes.io/projected/2daf4f2e-b620-424b-acb8-196b867647c9-kube-api-access-gn4rb\") pod \"payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz\" (UID: \"2daf4f2e-b620-424b-acb8-196b867647c9\") " pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:56:59.391113 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.391067 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:56:59.420432 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.420401 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:56:59.668392 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.668370 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5"] Sep 10 04:56:59.670041 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:56:59.670012 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62877871_b140_4e14_9aa5_5fd741a310a2.slice/crio-62a15f2ee9543c761a0905c1a09fc5880243e523f6cf4e091ee3efb843864cd8 WatchSource:0}: Error finding container 62a15f2ee9543c761a0905c1a09fc5880243e523f6cf4e091ee3efb843864cd8: Status 404 returned error can't find the container with id 62a15f2ee9543c761a0905c1a09fc5880243e523f6cf4e091ee3efb843864cd8 Sep 10 04:56:59.702856 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:56:59.702832 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz"] Sep 10 04:56:59.710793 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:56:59.710759 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2daf4f2e_b620_424b_acb8_196b867647c9.slice/crio-bab9e222dc3bd557192928609505541536866eef44ff79157a3e8277a37e7d86 WatchSource:0}: Error finding container bab9e222dc3bd557192928609505541536866eef44ff79157a3e8277a37e7d86: Status 404 returned error can't find the container with id bab9e222dc3bd557192928609505541536866eef44ff79157a3e8277a37e7d86 Sep 10 04:57:00.207680 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:00.207640 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" event={"ID":"2daf4f2e-b620-424b-acb8-196b867647c9","Type":"ContainerStarted","Data":"afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da"} Sep 10 04:57:00.208144 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:00.207744 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:57:00.208144 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:00.207769 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" event={"ID":"2daf4f2e-b620-424b-acb8-196b867647c9","Type":"ContainerStarted","Data":"bab9e222dc3bd557192928609505541536866eef44ff79157a3e8277a37e7d86"} Sep 10 04:57:00.209318 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:00.209291 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" event={"ID":"62877871-b140-4e14-9aa5-5fd741a310a2","Type":"ContainerStarted","Data":"b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0"} Sep 10 04:57:00.209428 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:00.209341 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" event={"ID":"62877871-b140-4e14-9aa5-5fd741a310a2","Type":"ContainerStarted","Data":"62a15f2ee9543c761a0905c1a09fc5880243e523f6cf4e091ee3efb843864cd8"} Sep 10 04:57:00.209428 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:00.209362 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:57:00.225236 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:00.225178 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" podStartSLOduration=1.225162352 podStartE2EDuration="1.225162352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:57:00.222642546 +0000 UTC m=+939.726475608" watchObservedRunningTime="2026-09-10 04:57:00.225162352 +0000 UTC m=+939.728995413" Sep 10 04:57:00.239067 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:00.239014 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" podStartSLOduration=1.238995131 podStartE2EDuration="1.238995131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:57:00.236579628 +0000 UTC m=+939.740412706" watchObservedRunningTime="2026-09-10 04:57:00.238995131 +0000 UTC m=+939.742828189" Sep 10 04:57:01.036830 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:01.036794 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="747e6ce6-9b54-45f9-b419-435f9b07ce7b" path="/var/lib/kubelet/pods/747e6ce6-9b54-45f9-b419-435f9b07ce7b/volumes" Sep 10 04:57:11.218472 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:11.218410 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:57:11.219076 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:11.218531 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:57:28.912638 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:28.912599 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz"] Sep 10 04:57:28.913153 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:28.912814 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" podUID="2daf4f2e-b620-424b-acb8-196b867647c9" containerName="payload-processing" containerID="cri-o://afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da" gracePeriod=30 Sep 10 04:57:28.921567 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:28.921082 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5"] Sep 10 04:57:28.921567 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:28.921326 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" podUID="62877871-b140-4e14-9aa5-5fd741a310a2" containerName="payload-pre-processing" containerID="cri-o://b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0" gracePeriod=30 Sep 10 04:57:29.221829 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.221799 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:57:29.226044 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.226016 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:57:29.328725 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.328686 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62877871-b140-4e14-9aa5-5fd741a310a2-plugins-config-volume\") pod \"62877871-b140-4e14-9aa5-5fd741a310a2\" (UID: \"62877871-b140-4e14-9aa5-5fd741a310a2\") " Sep 10 04:57:29.328928 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.328737 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daf4f2e-b620-424b-acb8-196b867647c9-plugins-config-volume\") pod \"2daf4f2e-b620-424b-acb8-196b867647c9\" (UID: \"2daf4f2e-b620-424b-acb8-196b867647c9\") " Sep 10 04:57:29.328928 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.328869 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn4rb\" (UniqueName: \"kubernetes.io/projected/2daf4f2e-b620-424b-acb8-196b867647c9-kube-api-access-gn4rb\") pod \"2daf4f2e-b620-424b-acb8-196b867647c9\" (UID: \"2daf4f2e-b620-424b-acb8-196b867647c9\") " Sep 10 04:57:29.328928 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.328897 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kpl8\" (UniqueName: \"kubernetes.io/projected/62877871-b140-4e14-9aa5-5fd741a310a2-kube-api-access-8kpl8\") pod \"62877871-b140-4e14-9aa5-5fd741a310a2\" (UID: \"62877871-b140-4e14-9aa5-5fd741a310a2\") " Sep 10 04:57:29.329110 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.329081 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62877871-b140-4e14-9aa5-5fd741a310a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62877871-b140-4e14-9aa5-5fd741a310a2" (UID: "62877871-b140-4e14-9aa5-5fd741a310a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:57:29.329272 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.329253 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62877871-b140-4e14-9aa5-5fd741a310a2-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:57:29.329604 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.329578 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2daf4f2e-b620-424b-acb8-196b867647c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2daf4f2e-b620-424b-acb8-196b867647c9" (UID: "2daf4f2e-b620-424b-acb8-196b867647c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:57:29.332058 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.332022 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2daf4f2e-b620-424b-acb8-196b867647c9-kube-api-access-gn4rb" (OuterVolumeSpecName: "kube-api-access-gn4rb") pod "2daf4f2e-b620-424b-acb8-196b867647c9" (UID: "2daf4f2e-b620-424b-acb8-196b867647c9"). InnerVolumeSpecName "kube-api-access-gn4rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:57:29.332542 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.332502 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62877871-b140-4e14-9aa5-5fd741a310a2-kube-api-access-8kpl8" (OuterVolumeSpecName: "kube-api-access-8kpl8") pod "62877871-b140-4e14-9aa5-5fd741a310a2" (UID: "62877871-b140-4e14-9aa5-5fd741a310a2"). InnerVolumeSpecName "kube-api-access-8kpl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:57:29.370312 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.370278 2572 generic.go:356] "Generic (PLEG): container finished" podID="2daf4f2e-b620-424b-acb8-196b867647c9" containerID="afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da" exitCode=0 Sep 10 04:57:29.370525 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.370320 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" Sep 10 04:57:29.370525 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.370374 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" event={"ID":"2daf4f2e-b620-424b-acb8-196b867647c9","Type":"ContainerDied","Data":"afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da"} Sep 10 04:57:29.370525 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.370438 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz" event={"ID":"2daf4f2e-b620-424b-acb8-196b867647c9","Type":"ContainerDied","Data":"bab9e222dc3bd557192928609505541536866eef44ff79157a3e8277a37e7d86"} Sep 10 04:57:29.370525 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.370486 2572 scope.go:117] "RemoveContainer" containerID="afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da" Sep 10 04:57:29.371771 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.371748 2572 generic.go:356] "Generic (PLEG): container finished" podID="62877871-b140-4e14-9aa5-5fd741a310a2" containerID="b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0" exitCode=0 Sep 10 04:57:29.371868 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.371786 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" Sep 10 04:57:29.371868 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.371821 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" event={"ID":"62877871-b140-4e14-9aa5-5fd741a310a2","Type":"ContainerDied","Data":"b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0"} Sep 10 04:57:29.371970 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.371867 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5" event={"ID":"62877871-b140-4e14-9aa5-5fd741a310a2","Type":"ContainerDied","Data":"62a15f2ee9543c761a0905c1a09fc5880243e523f6cf4e091ee3efb843864cd8"} Sep 10 04:57:29.389900 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.389879 2572 scope.go:117] "RemoveContainer" containerID="afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da" Sep 10 04:57:29.390268 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:57:29.390249 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da\": container with ID starting with afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da not found: ID does not exist" containerID="afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da" Sep 10 04:57:29.390328 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.390277 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da"} err="failed to get container status \"afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da\": rpc error: code = NotFound desc = could not find container \"afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da\": container with ID starting with afd9c710165e402c729418f73363750a0a9683e7fa40179555b9abcad25db9da not found: ID does not exist" Sep 10 04:57:29.390328 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.390295 2572 scope.go:117] "RemoveContainer" containerID="b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0" Sep 10 04:57:29.399368 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.399347 2572 scope.go:117] "RemoveContainer" containerID="b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0" Sep 10 04:57:29.399716 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:57:29.399699 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0\": container with ID starting with b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0 not found: ID does not exist" containerID="b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0" Sep 10 04:57:29.399784 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.399722 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0"} err="failed to get container status \"b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0\": rpc error: code = NotFound desc = could not find container \"b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0\": container with ID starting with b64a8c686ba52ffef5d2e538dda69e0cd6be0d0ef5986de9118e5bdae22da7a0 not found: ID does not exist" Sep 10 04:57:29.401308 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.401288 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz"] Sep 10 04:57:29.404528 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.404504 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ead8c2c2-54c4c6459c-rdlgz"] Sep 10 04:57:29.414915 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.414889 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5"] Sep 10 04:57:29.418721 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.418699 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ead8c2c2-567cd9c5c4-jgcn5"] Sep 10 04:57:29.430305 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.430241 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn4rb\" (UniqueName: \"kubernetes.io/projected/2daf4f2e-b620-424b-acb8-196b867647c9-kube-api-access-gn4rb\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:57:29.430305 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.430267 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kpl8\" (UniqueName: \"kubernetes.io/projected/62877871-b140-4e14-9aa5-5fd741a310a2-kube-api-access-8kpl8\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:57:29.430305 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:29.430278 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2daf4f2e-b620-424b-acb8-196b867647c9-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:57:31.037708 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:31.037669 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2daf4f2e-b620-424b-acb8-196b867647c9" path="/var/lib/kubelet/pods/2daf4f2e-b620-424b-acb8-196b867647c9/volumes" Sep 10 04:57:31.038104 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:31.038067 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62877871-b140-4e14-9aa5-5fd741a310a2" path="/var/lib/kubelet/pods/62877871-b140-4e14-9aa5-5fd741a310a2/volumes" Sep 10 04:57:32.898292 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.898250 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h"] Sep 10 04:57:32.898980 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.898957 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62877871-b140-4e14-9aa5-5fd741a310a2" containerName="payload-pre-processing" Sep 10 04:57:32.899038 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.898985 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="62877871-b140-4e14-9aa5-5fd741a310a2" containerName="payload-pre-processing" Sep 10 04:57:32.899038 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.898999 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2daf4f2e-b620-424b-acb8-196b867647c9" containerName="payload-processing" Sep 10 04:57:32.899038 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.899008 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2daf4f2e-b620-424b-acb8-196b867647c9" containerName="payload-processing" Sep 10 04:57:32.899151 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.899137 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="62877871-b140-4e14-9aa5-5fd741a310a2" containerName="payload-pre-processing" Sep 10 04:57:32.899191 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.899154 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2daf4f2e-b620-424b-acb8-196b867647c9" containerName="payload-processing" Sep 10 04:57:32.904636 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.904604 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:32.908416 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.908371 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f3cc69\"/\"test-model-f3cc69-kserve-self-signed-certs\"" Sep 10 04:57:32.908598 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.908436 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f3cc69\"/\"kube-root-ca.crt\"" Sep 10 04:57:32.908715 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.908694 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f3cc69\"/\"openshift-service-ca.crt\"" Sep 10 04:57:32.908835 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.908758 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f3cc69\"/\"default-dockercfg-7ckw4\"" Sep 10 04:57:32.913145 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.913113 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h"] Sep 10 04:57:32.963608 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.963574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-model-cache\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:32.963608 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.963618 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7ffr\" (UniqueName: \"kubernetes.io/projected/6bea8e93-1a89-4b3c-9652-77ec507783cc-kube-api-access-p7ffr\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:32.963854 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.963643 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bea8e93-1a89-4b3c-9652-77ec507783cc-tls-certs\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:32.963854 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.963674 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-home\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:32.963854 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.963729 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-dshm\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:32.963854 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:32.963745 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-tmp-dir\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.067016 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.066973 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7ffr\" (UniqueName: \"kubernetes.io/projected/6bea8e93-1a89-4b3c-9652-77ec507783cc-kube-api-access-p7ffr\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.067238 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.067033 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bea8e93-1a89-4b3c-9652-77ec507783cc-tls-certs\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.067238 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.067064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-home\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.067238 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.067151 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-dshm\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.067238 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.067174 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-tmp-dir\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.067476 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.067267 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-model-cache\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.067737 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.067709 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-model-cache\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.068938 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.068811 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-home\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.069049 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.069007 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-tmp-dir\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.073801 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.073763 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bea8e93-1a89-4b3c-9652-77ec507783cc-tls-certs\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.074212 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.074184 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-dshm\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.078623 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.078579 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7ffr\" (UniqueName: \"kubernetes.io/projected/6bea8e93-1a89-4b3c-9652-77ec507783cc-kube-api-access-p7ffr\") pod \"test-model-f3cc69-kserve-85684fbd85-vwm2h\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.217685 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.217587 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:33.494872 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:33.494843 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h"] Sep 10 04:57:33.497055 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:57:33.497026 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bea8e93_1a89_4b3c_9652_77ec507783cc.slice/crio-d3321e25f1664eec098f5b0fbb3043a8ea7f7870eb4326cd06784ae0bcfc3288 WatchSource:0}: Error finding container d3321e25f1664eec098f5b0fbb3043a8ea7f7870eb4326cd06784ae0bcfc3288: Status 404 returned error can't find the container with id d3321e25f1664eec098f5b0fbb3043a8ea7f7870eb4326cd06784ae0bcfc3288 Sep 10 04:57:34.402958 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:34.402921 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" event={"ID":"6bea8e93-1a89-4b3c-9652-77ec507783cc","Type":"ContainerStarted","Data":"2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7"} Sep 10 04:57:34.403366 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:34.402972 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" event={"ID":"6bea8e93-1a89-4b3c-9652-77ec507783cc","Type":"ContainerStarted","Data":"d3321e25f1664eec098f5b0fbb3043a8ea7f7870eb4326cd06784ae0bcfc3288"} Sep 10 04:57:34.419303 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:34.419250 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" podStartSLOduration=2.4192342829999998 podStartE2EDuration="2.419234283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:57:34.417908893 +0000 UTC m=+973.921741956" watchObservedRunningTime="2026-09-10 04:57:34.419234283 +0000 UTC m=+973.923067345" Sep 10 04:57:43.218117 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:43.218018 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:43.218117 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:43.218066 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:43.230771 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:43.230742 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:43.472060 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:43.471971 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 04:57:45.581764 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.581719 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb"] Sep 10 04:57:45.582290 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.581994 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" podUID="bdd993d4-cf3c-463c-9c34-0c223b74ae7a" containerName="payload-processing" containerID="cri-o://9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb" gracePeriod=30 Sep 10 04:57:45.587752 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.587716 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk"] Sep 10 04:57:45.588124 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.587960 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" podUID="bebfb768-521c-4c7a-aa3f-d526248d437f" containerName="payload-pre-processing" containerID="cri-o://012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34" gracePeriod=30 Sep 10 04:57:45.889495 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.889468 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:57:45.892653 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.892628 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:57:45.999231 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.999194 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndqv8\" (UniqueName: \"kubernetes.io/projected/bebfb768-521c-4c7a-aa3f-d526248d437f-kube-api-access-ndqv8\") pod \"bebfb768-521c-4c7a-aa3f-d526248d437f\" (UID: \"bebfb768-521c-4c7a-aa3f-d526248d437f\") " Sep 10 04:57:45.999436 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.999263 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-plugins-config-volume\") pod \"bdd993d4-cf3c-463c-9c34-0c223b74ae7a\" (UID: \"bdd993d4-cf3c-463c-9c34-0c223b74ae7a\") " Sep 10 04:57:45.999436 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.999382 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebfb768-521c-4c7a-aa3f-d526248d437f-plugins-config-volume\") pod \"bebfb768-521c-4c7a-aa3f-d526248d437f\" (UID: \"bebfb768-521c-4c7a-aa3f-d526248d437f\") " Sep 10 04:57:45.999436 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.999433 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc5nd\" (UniqueName: \"kubernetes.io/projected/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-kube-api-access-sc5nd\") pod \"bdd993d4-cf3c-463c-9c34-0c223b74ae7a\" (UID: \"bdd993d4-cf3c-463c-9c34-0c223b74ae7a\") " Sep 10 04:57:45.999766 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.999628 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bdd993d4-cf3c-463c-9c34-0c223b74ae7a" (UID: "bdd993d4-cf3c-463c-9c34-0c223b74ae7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:57:45.999820 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:45.999778 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebfb768-521c-4c7a-aa3f-d526248d437f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bebfb768-521c-4c7a-aa3f-d526248d437f" (UID: "bebfb768-521c-4c7a-aa3f-d526248d437f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:57:46.001457 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.001418 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebfb768-521c-4c7a-aa3f-d526248d437f-kube-api-access-ndqv8" (OuterVolumeSpecName: "kube-api-access-ndqv8") pod "bebfb768-521c-4c7a-aa3f-d526248d437f" (UID: "bebfb768-521c-4c7a-aa3f-d526248d437f"). InnerVolumeSpecName "kube-api-access-ndqv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:57:46.001639 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.001616 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-kube-api-access-sc5nd" (OuterVolumeSpecName: "kube-api-access-sc5nd") pod "bdd993d4-cf3c-463c-9c34-0c223b74ae7a" (UID: "bdd993d4-cf3c-463c-9c34-0c223b74ae7a"). InnerVolumeSpecName "kube-api-access-sc5nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:57:46.100323 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.100281 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebfb768-521c-4c7a-aa3f-d526248d437f-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:57:46.100323 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.100314 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc5nd\" (UniqueName: \"kubernetes.io/projected/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-kube-api-access-sc5nd\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:57:46.100323 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.100323 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndqv8\" (UniqueName: \"kubernetes.io/projected/bebfb768-521c-4c7a-aa3f-d526248d437f-kube-api-access-ndqv8\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:57:46.100323 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.100334 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdd993d4-cf3c-463c-9c34-0c223b74ae7a-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:57:46.477281 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.477241 2572 generic.go:356] "Generic (PLEG): container finished" podID="bdd993d4-cf3c-463c-9c34-0c223b74ae7a" containerID="9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb" exitCode=0 Sep 10 04:57:46.477518 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.477291 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" Sep 10 04:57:46.477518 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.477321 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" event={"ID":"bdd993d4-cf3c-463c-9c34-0c223b74ae7a","Type":"ContainerDied","Data":"9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb"} Sep 10 04:57:46.477518 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.477370 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb" event={"ID":"bdd993d4-cf3c-463c-9c34-0c223b74ae7a","Type":"ContainerDied","Data":"554014ea733f6eb668a4029be609809e79d846b46e32ae6da5d25bfc0512daf3"} Sep 10 04:57:46.477518 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.477399 2572 scope.go:117] "RemoveContainer" containerID="9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb" Sep 10 04:57:46.478892 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.478864 2572 generic.go:356] "Generic (PLEG): container finished" podID="bebfb768-521c-4c7a-aa3f-d526248d437f" containerID="012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34" exitCode=0 Sep 10 04:57:46.478985 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.478902 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" Sep 10 04:57:46.478985 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.478931 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" event={"ID":"bebfb768-521c-4c7a-aa3f-d526248d437f","Type":"ContainerDied","Data":"012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34"} Sep 10 04:57:46.478985 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.478978 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk" event={"ID":"bebfb768-521c-4c7a-aa3f-d526248d437f","Type":"ContainerDied","Data":"07b218a2a2f133833103f2442131adf159af0fdb5372b1785e9659571b8d8f16"} Sep 10 04:57:46.487974 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.487952 2572 scope.go:117] "RemoveContainer" containerID="9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb" Sep 10 04:57:46.488415 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:57:46.488388 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb\": container with ID starting with 9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb not found: ID does not exist" containerID="9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb" Sep 10 04:57:46.488525 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.488426 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb"} err="failed to get container status \"9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb\": rpc error: code = NotFound desc = could not find container \"9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb\": container with ID starting with 9b19e23e32fcbf8bce006f7bfb3d93a3a635c1daab926c9262b1c640844eaebb not found: ID does not exist" Sep 10 04:57:46.488525 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.488465 2572 scope.go:117] "RemoveContainer" containerID="012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34" Sep 10 04:57:46.497669 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.497641 2572 scope.go:117] "RemoveContainer" containerID="012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34" Sep 10 04:57:46.498014 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:57:46.497988 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34\": container with ID starting with 012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34 not found: ID does not exist" containerID="012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34" Sep 10 04:57:46.498089 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.498024 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34"} err="failed to get container status \"012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34\": rpc error: code = NotFound desc = could not find container \"012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34\": container with ID starting with 012f683e09386d3fb9fa32cf28dd035ae748ff86d45a67a230a2dfe131c03b34 not found: ID does not exist" Sep 10 04:57:46.502646 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.502610 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk"] Sep 10 04:57:46.505697 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.505660 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-53c734-7f7466cd7d-l5svk"] Sep 10 04:57:46.515878 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.515843 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb"] Sep 10 04:57:46.519225 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:46.519186 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-53c734-b4c7d47d7-hchbb"] Sep 10 04:57:47.036775 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:47.036742 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd993d4-cf3c-463c-9c34-0c223b74ae7a" path="/var/lib/kubelet/pods/bdd993d4-cf3c-463c-9c34-0c223b74ae7a/volumes" Sep 10 04:57:47.037159 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:57:47.037104 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bebfb768-521c-4c7a-aa3f-d526248d437f" path="/var/lib/kubelet/pods/bebfb768-521c-4c7a-aa3f-d526248d437f/volumes" Sep 10 04:58:33.601356 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.601320 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b"] Sep 10 04:58:33.602007 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.601982 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bebfb768-521c-4c7a-aa3f-d526248d437f" containerName="payload-pre-processing" Sep 10 04:58:33.602289 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.602272 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bebfb768-521c-4c7a-aa3f-d526248d437f" containerName="payload-pre-processing" Sep 10 04:58:33.602406 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.602393 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd993d4-cf3c-463c-9c34-0c223b74ae7a" containerName="payload-processing" Sep 10 04:58:33.602493 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.602472 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd993d4-cf3c-463c-9c34-0c223b74ae7a" containerName="payload-processing" Sep 10 04:58:33.602629 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.602615 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd993d4-cf3c-463c-9c34-0c223b74ae7a" containerName="payload-processing" Sep 10 04:58:33.602698 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.602641 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bebfb768-521c-4c7a-aa3f-d526248d437f" containerName="payload-pre-processing" Sep 10 04:58:33.607685 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.607662 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.610198 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.610175 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-ea9ba2-serving-cert\"" Sep 10 04:58:33.619036 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.619010 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b"] Sep 10 04:58:33.679702 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.679662 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0cbb13-e4c2-442b-98e6-eb0788a8866f-openshift-service-ca\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.679893 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.679718 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2qz8\" (UniqueName: \"kubernetes.io/projected/df0cbb13-e4c2-442b-98e6-eb0788a8866f-kube-api-access-g2qz8\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.679893 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.679766 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-maas-api-tls\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.679893 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.679829 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-metrics-certs\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.781292 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.781249 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0cbb13-e4c2-442b-98e6-eb0788a8866f-openshift-service-ca\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.781292 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.781302 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2qz8\" (UniqueName: \"kubernetes.io/projected/df0cbb13-e4c2-442b-98e6-eb0788a8866f-kube-api-access-g2qz8\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.781615 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.781434 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-maas-api-tls\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.781615 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.781542 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-metrics-certs\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.782061 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.782040 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0cbb13-e4c2-442b-98e6-eb0788a8866f-openshift-service-ca\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.783996 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.783966 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-metrics-certs\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.784102 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.784015 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-maas-api-tls\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.788997 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.788973 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2qz8\" (UniqueName: \"kubernetes.io/projected/df0cbb13-e4c2-442b-98e6-eb0788a8866f-kube-api-access-g2qz8\") pod \"maas-api-e2e-explicit-ea9ba2-56688bd47-q846b\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:33.920367 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:33.920256 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:34.196521 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.196493 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b"] Sep 10 04:58:34.198266 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:58:34.198238 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0cbb13_e4c2_442b_98e6_eb0788a8866f.slice/crio-f1a2ac3b04888f71a724f39aac4e3775c76ad10d10aa3208fb2dcad1065b9c30 WatchSource:0}: Error finding container f1a2ac3b04888f71a724f39aac4e3775c76ad10d10aa3208fb2dcad1065b9c30: Status 404 returned error can't find the container with id f1a2ac3b04888f71a724f39aac4e3775c76ad10d10aa3208fb2dcad1065b9c30 Sep 10 04:58:34.199775 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.199746 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 04:58:34.210488 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.210432 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t"] Sep 10 04:58:34.215551 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.215524 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:58:34.218339 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.218309 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-ea9ba2\"" Sep 10 04:58:34.218712 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.218695 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-ea9ba2-dockercfg-7lz8m\"" Sep 10 04:58:34.225505 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.225029 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t"] Sep 10 04:58:34.237503 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.237441 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h"] Sep 10 04:58:34.241388 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.241359 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:58:34.250134 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.249877 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h"] Sep 10 04:58:34.286923 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.286883 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564d4235-4925-4793-8881-f8cb3faee658-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t\" (UID: \"564d4235-4925-4793-8881-f8cb3faee658\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:58:34.287100 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.287005 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdpc4\" (UniqueName: \"kubernetes.io/projected/564d4235-4925-4793-8881-f8cb3faee658-kube-api-access-jdpc4\") pod \"payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t\" (UID: \"564d4235-4925-4793-8881-f8cb3faee658\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:58:34.388164 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.388122 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7a73410-cfb0-4a55-91c1-ca6d8b594408-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h\" (UID: \"a7a73410-cfb0-4a55-91c1-ca6d8b594408\") " pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:58:34.388164 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.388170 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdpc4\" (UniqueName: \"kubernetes.io/projected/564d4235-4925-4793-8881-f8cb3faee658-kube-api-access-jdpc4\") pod \"payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t\" (UID: \"564d4235-4925-4793-8881-f8cb3faee658\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:58:34.388392 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.388313 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkd6l\" (UniqueName: \"kubernetes.io/projected/a7a73410-cfb0-4a55-91c1-ca6d8b594408-kube-api-access-jkd6l\") pod \"payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h\" (UID: \"a7a73410-cfb0-4a55-91c1-ca6d8b594408\") " pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:58:34.388392 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.388354 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564d4235-4925-4793-8881-f8cb3faee658-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t\" (UID: \"564d4235-4925-4793-8881-f8cb3faee658\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:58:34.389034 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.389016 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564d4235-4925-4793-8881-f8cb3faee658-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t\" (UID: \"564d4235-4925-4793-8881-f8cb3faee658\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:58:34.396177 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.396150 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdpc4\" (UniqueName: \"kubernetes.io/projected/564d4235-4925-4793-8881-f8cb3faee658-kube-api-access-jdpc4\") pod \"payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t\" (UID: \"564d4235-4925-4793-8881-f8cb3faee658\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:58:34.489534 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.489492 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7a73410-cfb0-4a55-91c1-ca6d8b594408-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h\" (UID: \"a7a73410-cfb0-4a55-91c1-ca6d8b594408\") " pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:58:34.489709 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.489592 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkd6l\" (UniqueName: \"kubernetes.io/projected/a7a73410-cfb0-4a55-91c1-ca6d8b594408-kube-api-access-jkd6l\") pod \"payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h\" (UID: \"a7a73410-cfb0-4a55-91c1-ca6d8b594408\") " pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:58:34.490169 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.490147 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7a73410-cfb0-4a55-91c1-ca6d8b594408-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h\" (UID: \"a7a73410-cfb0-4a55-91c1-ca6d8b594408\") " pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:58:34.498257 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.498231 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkd6l\" (UniqueName: \"kubernetes.io/projected/a7a73410-cfb0-4a55-91c1-ca6d8b594408-kube-api-access-jkd6l\") pod \"payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h\" (UID: \"a7a73410-cfb0-4a55-91c1-ca6d8b594408\") " pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:58:34.530043 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.530001 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:58:34.554616 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.554581 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:58:34.747133 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.747052 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" event={"ID":"df0cbb13-e4c2-442b-98e6-eb0788a8866f","Type":"ContainerStarted","Data":"5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3"} Sep 10 04:58:34.747133 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.747117 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:34.747572 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.747142 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" event={"ID":"df0cbb13-e4c2-442b-98e6-eb0788a8866f","Type":"ContainerStarted","Data":"f1a2ac3b04888f71a724f39aac4e3775c76ad10d10aa3208fb2dcad1065b9c30"} Sep 10 04:58:34.767053 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.766996 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" podStartSLOduration=1.288591043 podStartE2EDuration="1.76697806s" totalImagesPullingTime="478.387017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:58:34.764148927 +0000 UTC m=+1034.267982010" watchObservedRunningTime="2026-09-10 04:58:34.76697806 +0000 UTC m=+1034.270811120" Sep 10 04:58:34.803985 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.803955 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t"] Sep 10 04:58:34.805291 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:58:34.805261 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564d4235_4925_4793_8881_f8cb3faee658.slice/crio-91f2a5c4a5472aa725e01640db4d1f544199cc6bc8f084d04a1e81ab536d423a WatchSource:0}: Error finding container 91f2a5c4a5472aa725e01640db4d1f544199cc6bc8f084d04a1e81ab536d423a: Status 404 returned error can't find the container with id 91f2a5c4a5472aa725e01640db4d1f544199cc6bc8f084d04a1e81ab536d423a Sep 10 04:58:34.838313 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:34.838258 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h"] Sep 10 04:58:34.844820 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:58:34.844793 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a73410_cfb0_4a55_91c1_ca6d8b594408.slice/crio-4539383a2acf3973c468da03903e107b60bee113b38bbce4e09afaff40172b47 WatchSource:0}: Error finding container 4539383a2acf3973c468da03903e107b60bee113b38bbce4e09afaff40172b47: Status 404 returned error can't find the container with id 4539383a2acf3973c468da03903e107b60bee113b38bbce4e09afaff40172b47 Sep 10 04:58:35.753911 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:35.753873 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" event={"ID":"564d4235-4925-4793-8881-f8cb3faee658","Type":"ContainerStarted","Data":"1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619"} Sep 10 04:58:35.754379 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:35.753930 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:58:35.754379 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:35.753954 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" event={"ID":"564d4235-4925-4793-8881-f8cb3faee658","Type":"ContainerStarted","Data":"91f2a5c4a5472aa725e01640db4d1f544199cc6bc8f084d04a1e81ab536d423a"} Sep 10 04:58:35.755508 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:35.755481 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" event={"ID":"a7a73410-cfb0-4a55-91c1-ca6d8b594408","Type":"ContainerStarted","Data":"d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90"} Sep 10 04:58:35.755631 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:35.755604 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:58:35.755631 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:35.755626 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" event={"ID":"a7a73410-cfb0-4a55-91c1-ca6d8b594408","Type":"ContainerStarted","Data":"4539383a2acf3973c468da03903e107b60bee113b38bbce4e09afaff40172b47"} Sep 10 04:58:35.770911 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:35.770859 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" podStartSLOduration=1.7708422879999999 podStartE2EDuration="1.770842288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:58:35.768339113 +0000 UTC m=+1035.272172172" watchObservedRunningTime="2026-09-10 04:58:35.770842288 +0000 UTC m=+1035.274675349" Sep 10 04:58:35.785538 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:35.785483 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" podStartSLOduration=1.7854691040000001 podStartE2EDuration="1.785469104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:58:35.783755008 +0000 UTC m=+1035.287588081" watchObservedRunningTime="2026-09-10 04:58:35.785469104 +0000 UTC m=+1035.289302165" Sep 10 04:58:40.761641 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:40.761607 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:58:46.765688 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:46.765653 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:58:46.766175 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:58:46.766155 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:59:11.672654 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.672616 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx"] Sep 10 04:59:11.677586 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.677566 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:11.680002 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.679981 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-aa81ae79\"" Sep 10 04:59:11.687279 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.687254 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx"] Sep 10 04:59:11.704887 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.704832 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx"] Sep 10 04:59:11.708694 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.708667 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:11.711297 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.711273 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-aa81ae79-dockercfg-j9gst\"" Sep 10 04:59:11.719209 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.719181 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx"] Sep 10 04:59:11.779243 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.779204 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c7c7\" (UniqueName: \"kubernetes.io/projected/8876af21-3057-4be2-8ea4-ae681b457ccc-kube-api-access-4c7c7\") pod \"payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx\" (UID: \"8876af21-3057-4be2-8ea4-ae681b457ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:11.779423 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.779252 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8876af21-3057-4be2-8ea4-ae681b457ccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx\" (UID: \"8876af21-3057-4be2-8ea4-ae681b457ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:11.880338 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.880300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5d6r\" (UniqueName: \"kubernetes.io/projected/f2540175-2fd9-4221-a446-9b965d665c0f-kube-api-access-v5d6r\") pod \"payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx\" (UID: \"f2540175-2fd9-4221-a446-9b965d665c0f\") " pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:11.880527 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.880429 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c7c7\" (UniqueName: \"kubernetes.io/projected/8876af21-3057-4be2-8ea4-ae681b457ccc-kube-api-access-4c7c7\") pod \"payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx\" (UID: \"8876af21-3057-4be2-8ea4-ae681b457ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:11.880527 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.880476 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2540175-2fd9-4221-a446-9b965d665c0f-plugins-config-volume\") pod \"payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx\" (UID: \"f2540175-2fd9-4221-a446-9b965d665c0f\") " pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:11.880527 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.880500 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8876af21-3057-4be2-8ea4-ae681b457ccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx\" (UID: \"8876af21-3057-4be2-8ea4-ae681b457ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:11.881260 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.881230 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8876af21-3057-4be2-8ea4-ae681b457ccc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx\" (UID: \"8876af21-3057-4be2-8ea4-ae681b457ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:11.888403 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.888374 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c7c7\" (UniqueName: \"kubernetes.io/projected/8876af21-3057-4be2-8ea4-ae681b457ccc-kube-api-access-4c7c7\") pod \"payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx\" (UID: \"8876af21-3057-4be2-8ea4-ae681b457ccc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:11.982154 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.982040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2540175-2fd9-4221-a446-9b965d665c0f-plugins-config-volume\") pod \"payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx\" (UID: \"f2540175-2fd9-4221-a446-9b965d665c0f\") " pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:11.982154 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.982127 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5d6r\" (UniqueName: \"kubernetes.io/projected/f2540175-2fd9-4221-a446-9b965d665c0f-kube-api-access-v5d6r\") pod \"payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx\" (UID: \"f2540175-2fd9-4221-a446-9b965d665c0f\") " pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:11.982835 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.982814 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2540175-2fd9-4221-a446-9b965d665c0f-plugins-config-volume\") pod \"payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx\" (UID: \"f2540175-2fd9-4221-a446-9b965d665c0f\") " pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:11.990019 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.989986 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5d6r\" (UniqueName: \"kubernetes.io/projected/f2540175-2fd9-4221-a446-9b965d665c0f-kube-api-access-v5d6r\") pod \"payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx\" (UID: \"f2540175-2fd9-4221-a446-9b965d665c0f\") " pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:11.990788 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:11.990764 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:12.020339 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:12.020292 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:12.271540 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:12.271511 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx"] Sep 10 04:59:12.272801 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:59:12.272777 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8876af21_3057_4be2_8ea4_ae681b457ccc.slice/crio-0a7c020edefca8053bc7ec4dbe31477d1d2ae730390f21c2f5d3421c6ad056ca WatchSource:0}: Error finding container 0a7c020edefca8053bc7ec4dbe31477d1d2ae730390f21c2f5d3421c6ad056ca: Status 404 returned error can't find the container with id 0a7c020edefca8053bc7ec4dbe31477d1d2ae730390f21c2f5d3421c6ad056ca Sep 10 04:59:12.306173 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:12.306069 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx"] Sep 10 04:59:12.307496 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:59:12.307432 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2540175_2fd9_4221_a446_9b965d665c0f.slice/crio-52129b4f7c3bada9a705bfe372a3940923e7d878b5b1ef355953a90d980d5604 WatchSource:0}: Error finding container 52129b4f7c3bada9a705bfe372a3940923e7d878b5b1ef355953a90d980d5604: Status 404 returned error can't find the container with id 52129b4f7c3bada9a705bfe372a3940923e7d878b5b1ef355953a90d980d5604 Sep 10 04:59:12.961527 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:12.961477 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" event={"ID":"f2540175-2fd9-4221-a446-9b965d665c0f","Type":"ContainerStarted","Data":"5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c"} Sep 10 04:59:12.962021 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:12.961596 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:12.962021 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:12.961625 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" event={"ID":"f2540175-2fd9-4221-a446-9b965d665c0f","Type":"ContainerStarted","Data":"52129b4f7c3bada9a705bfe372a3940923e7d878b5b1ef355953a90d980d5604"} Sep 10 04:59:12.963331 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:12.963302 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" event={"ID":"8876af21-3057-4be2-8ea4-ae681b457ccc","Type":"ContainerStarted","Data":"c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea"} Sep 10 04:59:12.963522 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:12.963364 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" event={"ID":"8876af21-3057-4be2-8ea4-ae681b457ccc","Type":"ContainerStarted","Data":"0a7c020edefca8053bc7ec4dbe31477d1d2ae730390f21c2f5d3421c6ad056ca"} Sep 10 04:59:12.963522 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:12.963393 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:12.980188 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:12.980112 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" podStartSLOduration=1.980085973 podStartE2EDuration="1.980085973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:59:12.978252032 +0000 UTC m=+1072.482085093" watchObservedRunningTime="2026-09-10 04:59:12.980085973 +0000 UTC m=+1072.483919037" Sep 10 04:59:12.993228 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:12.993171 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" podStartSLOduration=1.993155974 podStartE2EDuration="1.993155974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:59:12.993091291 +0000 UTC m=+1072.496924351" watchObservedRunningTime="2026-09-10 04:59:12.993155974 +0000 UTC m=+1072.496989034" Sep 10 04:59:23.969999 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:23.969963 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:23.970512 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:23.970232 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:35.300031 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.299988 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b"] Sep 10 04:59:35.300575 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.300244 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" podUID="df0cbb13-e4c2-442b-98e6-eb0788a8866f" containerName="maas-api" containerID="cri-o://5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3" gracePeriod=30 Sep 10 04:59:35.368366 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.368324 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h"] Sep 10 04:59:35.368695 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.368660 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" podUID="a7a73410-cfb0-4a55-91c1-ca6d8b594408" containerName="payload-processing" containerID="cri-o://d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90" gracePeriod=30 Sep 10 04:59:35.374844 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.374793 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t"] Sep 10 04:59:35.375120 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.375073 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" podUID="564d4235-4925-4793-8881-f8cb3faee658" containerName="payload-pre-processing" containerID="cri-o://1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619" gracePeriod=30 Sep 10 04:59:35.687164 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.687009 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:59:35.701643 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.701614 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:59:35.712873 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.712405 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:59:35.845177 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.845074 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564d4235-4925-4793-8881-f8cb3faee658-plugins-config-volume\") pod \"564d4235-4925-4793-8881-f8cb3faee658\" (UID: \"564d4235-4925-4793-8881-f8cb3faee658\") " Sep 10 04:59:35.845177 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.845162 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2qz8\" (UniqueName: \"kubernetes.io/projected/df0cbb13-e4c2-442b-98e6-eb0788a8866f-kube-api-access-g2qz8\") pod \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " Sep 10 04:59:35.845399 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.845206 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdpc4\" (UniqueName: \"kubernetes.io/projected/564d4235-4925-4793-8881-f8cb3faee658-kube-api-access-jdpc4\") pod \"564d4235-4925-4793-8881-f8cb3faee658\" (UID: \"564d4235-4925-4793-8881-f8cb3faee658\") " Sep 10 04:59:35.845399 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.845248 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkd6l\" (UniqueName: \"kubernetes.io/projected/a7a73410-cfb0-4a55-91c1-ca6d8b594408-kube-api-access-jkd6l\") pod \"a7a73410-cfb0-4a55-91c1-ca6d8b594408\" (UID: \"a7a73410-cfb0-4a55-91c1-ca6d8b594408\") " Sep 10 04:59:35.845399 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.845287 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-metrics-certs\") pod \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " Sep 10 04:59:35.845399 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.845322 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-maas-api-tls\") pod \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " Sep 10 04:59:35.845399 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.845348 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0cbb13-e4c2-442b-98e6-eb0788a8866f-openshift-service-ca\") pod \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\" (UID: \"df0cbb13-e4c2-442b-98e6-eb0788a8866f\") " Sep 10 04:59:35.845399 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.845395 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7a73410-cfb0-4a55-91c1-ca6d8b594408-plugins-config-volume\") pod \"a7a73410-cfb0-4a55-91c1-ca6d8b594408\" (UID: \"a7a73410-cfb0-4a55-91c1-ca6d8b594408\") " Sep 10 04:59:35.845788 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.845741 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0cbb13-e4c2-442b-98e6-eb0788a8866f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df0cbb13-e4c2-442b-98e6-eb0788a8866f" (UID: "df0cbb13-e4c2-442b-98e6-eb0788a8866f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:59:35.846033 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.845967 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0cbb13-e4c2-442b-98e6-eb0788a8866f-openshift-service-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:35.846166 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.846064 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a73410-cfb0-4a55-91c1-ca6d8b594408-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7a73410-cfb0-4a55-91c1-ca6d8b594408" (UID: "a7a73410-cfb0-4a55-91c1-ca6d8b594408"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:59:35.846166 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.846090 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564d4235-4925-4793-8881-f8cb3faee658-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "564d4235-4925-4793-8881-f8cb3faee658" (UID: "564d4235-4925-4793-8881-f8cb3faee658"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:59:35.847601 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.847571 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df0cbb13-e4c2-442b-98e6-eb0788a8866f" (UID: "df0cbb13-e4c2-442b-98e6-eb0788a8866f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:59:35.848090 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.848059 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df0cbb13-e4c2-442b-98e6-eb0788a8866f" (UID: "df0cbb13-e4c2-442b-98e6-eb0788a8866f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 04:59:35.848090 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.848071 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0cbb13-e4c2-442b-98e6-eb0788a8866f-kube-api-access-g2qz8" (OuterVolumeSpecName: "kube-api-access-g2qz8") pod "df0cbb13-e4c2-442b-98e6-eb0788a8866f" (UID: "df0cbb13-e4c2-442b-98e6-eb0788a8866f"). InnerVolumeSpecName "kube-api-access-g2qz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:59:35.848185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.848084 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564d4235-4925-4793-8881-f8cb3faee658-kube-api-access-jdpc4" (OuterVolumeSpecName: "kube-api-access-jdpc4") pod "564d4235-4925-4793-8881-f8cb3faee658" (UID: "564d4235-4925-4793-8881-f8cb3faee658"). InnerVolumeSpecName "kube-api-access-jdpc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:59:35.848185 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.848081 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a73410-cfb0-4a55-91c1-ca6d8b594408-kube-api-access-jkd6l" (OuterVolumeSpecName: "kube-api-access-jkd6l") pod "a7a73410-cfb0-4a55-91c1-ca6d8b594408" (UID: "a7a73410-cfb0-4a55-91c1-ca6d8b594408"). InnerVolumeSpecName "kube-api-access-jkd6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:59:35.947078 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.947040 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564d4235-4925-4793-8881-f8cb3faee658-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:35.947078 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.947075 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2qz8\" (UniqueName: \"kubernetes.io/projected/df0cbb13-e4c2-442b-98e6-eb0788a8866f-kube-api-access-g2qz8\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:35.947078 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.947084 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdpc4\" (UniqueName: \"kubernetes.io/projected/564d4235-4925-4793-8881-f8cb3faee658-kube-api-access-jdpc4\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:35.947330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.947093 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkd6l\" (UniqueName: \"kubernetes.io/projected/a7a73410-cfb0-4a55-91c1-ca6d8b594408-kube-api-access-jkd6l\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:35.947330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.947103 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:35.947330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.947111 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df0cbb13-e4c2-442b-98e6-eb0788a8866f-maas-api-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:35.947330 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:35.947120 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7a73410-cfb0-4a55-91c1-ca6d8b594408-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:36.092133 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.092097 2572 generic.go:356] "Generic (PLEG): container finished" podID="564d4235-4925-4793-8881-f8cb3faee658" containerID="1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619" exitCode=0 Sep 10 04:59:36.092326 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.092142 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" Sep 10 04:59:36.092326 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.092181 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" event={"ID":"564d4235-4925-4793-8881-f8cb3faee658","Type":"ContainerDied","Data":"1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619"} Sep 10 04:59:36.092326 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.092242 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t" event={"ID":"564d4235-4925-4793-8881-f8cb3faee658","Type":"ContainerDied","Data":"91f2a5c4a5472aa725e01640db4d1f544199cc6bc8f084d04a1e81ab536d423a"} Sep 10 04:59:36.092326 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.092276 2572 scope.go:117] "RemoveContainer" containerID="1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619" Sep 10 04:59:36.093652 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.093632 2572 generic.go:356] "Generic (PLEG): container finished" podID="df0cbb13-e4c2-442b-98e6-eb0788a8866f" containerID="5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3" exitCode=0 Sep 10 04:59:36.093761 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.093662 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" event={"ID":"df0cbb13-e4c2-442b-98e6-eb0788a8866f","Type":"ContainerDied","Data":"5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3"} Sep 10 04:59:36.093761 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.093721 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" event={"ID":"df0cbb13-e4c2-442b-98e6-eb0788a8866f","Type":"ContainerDied","Data":"f1a2ac3b04888f71a724f39aac4e3775c76ad10d10aa3208fb2dcad1065b9c30"} Sep 10 04:59:36.093761 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.093679 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b" Sep 10 04:59:36.095372 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.095219 2572 generic.go:356] "Generic (PLEG): container finished" podID="a7a73410-cfb0-4a55-91c1-ca6d8b594408" containerID="d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90" exitCode=0 Sep 10 04:59:36.095372 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.095253 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" event={"ID":"a7a73410-cfb0-4a55-91c1-ca6d8b594408","Type":"ContainerDied","Data":"d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90"} Sep 10 04:59:36.095372 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.095298 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" event={"ID":"a7a73410-cfb0-4a55-91c1-ca6d8b594408","Type":"ContainerDied","Data":"4539383a2acf3973c468da03903e107b60bee113b38bbce4e09afaff40172b47"} Sep 10 04:59:36.095372 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.095299 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h" Sep 10 04:59:36.103396 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.103377 2572 scope.go:117] "RemoveContainer" containerID="1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619" Sep 10 04:59:36.103767 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:59:36.103746 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619\": container with ID starting with 1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619 not found: ID does not exist" containerID="1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619" Sep 10 04:59:36.103866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.103775 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619"} err="failed to get container status \"1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619\": rpc error: code = NotFound desc = could not find container \"1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619\": container with ID starting with 1df58c14ee71225dddc5eba23ca1cb7fd3f1566e10e02fc136275c9baa387619 not found: ID does not exist" Sep 10 04:59:36.103866 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.103792 2572 scope.go:117] "RemoveContainer" containerID="5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3" Sep 10 04:59:36.112637 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.112614 2572 scope.go:117] "RemoveContainer" containerID="5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3" Sep 10 04:59:36.113078 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:59:36.112949 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3\": container with ID starting with 5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3 not found: ID does not exist" containerID="5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3" Sep 10 04:59:36.113122 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.113088 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3"} err="failed to get container status \"5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3\": rpc error: code = NotFound desc = could not find container \"5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3\": container with ID starting with 5c671311a87aa0e81344587763a5701feeec8d3038f30ab1ab337548d8225ad3 not found: ID does not exist" Sep 10 04:59:36.113178 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.113137 2572 scope.go:117] "RemoveContainer" containerID="d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90" Sep 10 04:59:36.116686 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.116664 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t"] Sep 10 04:59:36.119934 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.119906 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ea9ba2-6bc57c46c4-6j46t"] Sep 10 04:59:36.122678 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.122660 2572 scope.go:117] "RemoveContainer" containerID="d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90" Sep 10 04:59:36.123019 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:59:36.122997 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90\": container with ID starting with d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90 not found: ID does not exist" containerID="d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90" Sep 10 04:59:36.123087 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.123026 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90"} err="failed to get container status \"d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90\": rpc error: code = NotFound desc = could not find container \"d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90\": container with ID starting with d84796e4170d9c6a38514a78e235f90bee3cdafd3641ded5591f97ad71b8af90 not found: ID does not exist" Sep 10 04:59:36.133984 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.133956 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b"] Sep 10 04:59:36.139305 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.139279 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ea9ba2-56688bd47-q846b"] Sep 10 04:59:36.150809 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.150777 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h"] Sep 10 04:59:36.154750 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:36.154708 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ea9ba2-764b5f9b85-4m89h"] Sep 10 04:59:37.038215 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:37.038181 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="564d4235-4925-4793-8881-f8cb3faee658" path="/var/lib/kubelet/pods/564d4235-4925-4793-8881-f8cb3faee658/volumes" Sep 10 04:59:37.038614 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:37.038586 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a73410-cfb0-4a55-91c1-ca6d8b594408" path="/var/lib/kubelet/pods/a7a73410-cfb0-4a55-91c1-ca6d8b594408/volumes" Sep 10 04:59:37.038944 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:37.038928 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df0cbb13-e4c2-442b-98e6-eb0788a8866f" path="/var/lib/kubelet/pods/df0cbb13-e4c2-442b-98e6-eb0788a8866f/volumes" Sep 10 04:59:43.487351 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.487312 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b"] Sep 10 04:59:43.487994 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.487975 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0cbb13-e4c2-442b-98e6-eb0788a8866f" containerName="maas-api" Sep 10 04:59:43.487994 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.487995 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0cbb13-e4c2-442b-98e6-eb0788a8866f" containerName="maas-api" Sep 10 04:59:43.488115 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.488005 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a73410-cfb0-4a55-91c1-ca6d8b594408" containerName="payload-processing" Sep 10 04:59:43.488115 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.488011 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a73410-cfb0-4a55-91c1-ca6d8b594408" containerName="payload-processing" Sep 10 04:59:43.488115 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.488019 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="564d4235-4925-4793-8881-f8cb3faee658" containerName="payload-pre-processing" Sep 10 04:59:43.488115 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.488026 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="564d4235-4925-4793-8881-f8cb3faee658" containerName="payload-pre-processing" Sep 10 04:59:43.488115 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.488093 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0cbb13-e4c2-442b-98e6-eb0788a8866f" containerName="maas-api" Sep 10 04:59:43.488115 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.488103 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="564d4235-4925-4793-8881-f8cb3faee658" containerName="payload-pre-processing" Sep 10 04:59:43.488115 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.488111 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a73410-cfb0-4a55-91c1-ca6d8b594408" containerName="payload-processing" Sep 10 04:59:43.491515 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.491494 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.494221 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.494199 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-3e5c28-openshift-default-dockercfg-xqfz6\"" Sep 10 04:59:43.504711 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.504677 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b"] Sep 10 04:59:43.625837 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.625797 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-envoy\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.625837 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.625839 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-socket\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.626060 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.625861 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-data\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.626060 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.625941 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-istio-token\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.626060 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.625999 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a170ab36-72e8-4563-800e-57860915f26b-istio-podinfo\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.626060 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.626020 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a170ab36-72e8-4563-800e-57860915f26b-istiod-ca-cert\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.626060 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.626039 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-credential-socket\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.626245 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.626125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cdlz\" (UniqueName: \"kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-kube-api-access-4cdlz\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.626245 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.626201 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-certs\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.726970 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.726928 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-certs\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727148 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.726992 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-envoy\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727148 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.727019 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-socket\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727148 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.727039 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-data\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727148 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.727077 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-istio-token\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727299 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.727231 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a170ab36-72e8-4563-800e-57860915f26b-istio-podinfo\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727299 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.727273 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a170ab36-72e8-4563-800e-57860915f26b-istiod-ca-cert\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727404 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.727304 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-credential-socket\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727404 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.727378 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cdlz\" (UniqueName: \"kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-kube-api-access-4cdlz\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727404 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.727394 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-certs\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727620 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.727443 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-socket\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727620 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.727590 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-data\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.727765 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.727739 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-credential-socket\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.728051 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.728024 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a170ab36-72e8-4563-800e-57860915f26b-istiod-ca-cert\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.729368 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.729344 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-envoy\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.729969 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.729949 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a170ab36-72e8-4563-800e-57860915f26b-istio-podinfo\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.739610 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.739546 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cdlz\" (UniqueName: \"kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-kube-api-access-4cdlz\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.743099 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.743075 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-istio-token\") pod \"e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:43.802971 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:43.802934 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:44.085729 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:44.085698 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b"] Sep 10 04:59:44.088063 ip-10-0-136-213 kubenswrapper[2572]: W0910 04:59:44.088035 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda170ab36_72e8_4563_800e_57860915f26b.slice/crio-ec1e88f001b39ddd6b9907c29d17003348f31ec53be82fa7491cac2c71c3b12c WatchSource:0}: Error finding container ec1e88f001b39ddd6b9907c29d17003348f31ec53be82fa7491cac2c71c3b12c: Status 404 returned error can't find the container with id ec1e88f001b39ddd6b9907c29d17003348f31ec53be82fa7491cac2c71c3b12c Sep 10 04:59:44.090356 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:44.090324 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:59:44.090423 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:44.090392 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:59:44.090492 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:44.090461 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 04:59:44.144003 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:44.143968 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" event={"ID":"a170ab36-72e8-4563-800e-57860915f26b","Type":"ContainerStarted","Data":"ec1e88f001b39ddd6b9907c29d17003348f31ec53be82fa7491cac2c71c3b12c"} Sep 10 04:59:45.151356 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:45.151321 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" event={"ID":"a170ab36-72e8-4563-800e-57860915f26b","Type":"ContainerStarted","Data":"a36c8f839bab64e0bc3893f092b159aa1f8b20ec9ad852bcc2982347ed605235"} Sep 10 04:59:45.171466 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:45.171388 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" podStartSLOduration=2.17136606 podStartE2EDuration="2.17136606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 04:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 04:59:45.169104942 +0000 UTC m=+1104.672938021" watchObservedRunningTime="2026-09-10 04:59:45.17136606 +0000 UTC m=+1104.675199122" Sep 10 04:59:45.803784 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:45.803749 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:45.809116 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:45.809086 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:46.157427 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:46.157340 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:46.158467 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:46.158423 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 04:59:53.207408 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.207371 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx"] Sep 10 04:59:53.208219 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.208163 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" podUID="f2540175-2fd9-4221-a446-9b965d665c0f" containerName="payload-processing" containerID="cri-o://5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c" gracePeriod=30 Sep 10 04:59:53.210608 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.210579 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx"] Sep 10 04:59:53.211260 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.210879 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" podUID="8876af21-3057-4be2-8ea4-ae681b457ccc" containerName="payload-pre-processing" containerID="cri-o://c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea" gracePeriod=30 Sep 10 04:59:53.546537 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.546511 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:53.549948 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.549921 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:53.622147 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.622098 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2540175-2fd9-4221-a446-9b965d665c0f-plugins-config-volume\") pod \"f2540175-2fd9-4221-a446-9b965d665c0f\" (UID: \"f2540175-2fd9-4221-a446-9b965d665c0f\") " Sep 10 04:59:53.622338 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.622229 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c7c7\" (UniqueName: \"kubernetes.io/projected/8876af21-3057-4be2-8ea4-ae681b457ccc-kube-api-access-4c7c7\") pod \"8876af21-3057-4be2-8ea4-ae681b457ccc\" (UID: \"8876af21-3057-4be2-8ea4-ae681b457ccc\") " Sep 10 04:59:53.622338 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.622261 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5d6r\" (UniqueName: \"kubernetes.io/projected/f2540175-2fd9-4221-a446-9b965d665c0f-kube-api-access-v5d6r\") pod \"f2540175-2fd9-4221-a446-9b965d665c0f\" (UID: \"f2540175-2fd9-4221-a446-9b965d665c0f\") " Sep 10 04:59:53.622338 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.622305 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8876af21-3057-4be2-8ea4-ae681b457ccc-plugins-config-volume\") pod \"8876af21-3057-4be2-8ea4-ae681b457ccc\" (UID: \"8876af21-3057-4be2-8ea4-ae681b457ccc\") " Sep 10 04:59:53.622728 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.622569 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2540175-2fd9-4221-a446-9b965d665c0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2540175-2fd9-4221-a446-9b965d665c0f" (UID: "f2540175-2fd9-4221-a446-9b965d665c0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:59:53.622830 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.622749 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8876af21-3057-4be2-8ea4-ae681b457ccc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8876af21-3057-4be2-8ea4-ae681b457ccc" (UID: "8876af21-3057-4be2-8ea4-ae681b457ccc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 04:59:53.624676 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.624648 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2540175-2fd9-4221-a446-9b965d665c0f-kube-api-access-v5d6r" (OuterVolumeSpecName: "kube-api-access-v5d6r") pod "f2540175-2fd9-4221-a446-9b965d665c0f" (UID: "f2540175-2fd9-4221-a446-9b965d665c0f"). InnerVolumeSpecName "kube-api-access-v5d6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:59:53.624782 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.624742 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8876af21-3057-4be2-8ea4-ae681b457ccc-kube-api-access-4c7c7" (OuterVolumeSpecName: "kube-api-access-4c7c7") pod "8876af21-3057-4be2-8ea4-ae681b457ccc" (UID: "8876af21-3057-4be2-8ea4-ae681b457ccc"). InnerVolumeSpecName "kube-api-access-4c7c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 04:59:53.723972 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.723927 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2540175-2fd9-4221-a446-9b965d665c0f-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:53.723972 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.723966 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c7c7\" (UniqueName: \"kubernetes.io/projected/8876af21-3057-4be2-8ea4-ae681b457ccc-kube-api-access-4c7c7\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:53.723972 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.723975 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5d6r\" (UniqueName: \"kubernetes.io/projected/f2540175-2fd9-4221-a446-9b965d665c0f-kube-api-access-v5d6r\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:53.723972 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:53.723984 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8876af21-3057-4be2-8ea4-ae681b457ccc-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 04:59:54.209938 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.209903 2572 generic.go:356] "Generic (PLEG): container finished" podID="8876af21-3057-4be2-8ea4-ae681b457ccc" containerID="c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea" exitCode=0 Sep 10 04:59:54.210425 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.209964 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" Sep 10 04:59:54.210425 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.209992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" event={"ID":"8876af21-3057-4be2-8ea4-ae681b457ccc","Type":"ContainerDied","Data":"c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea"} Sep 10 04:59:54.210425 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.210039 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx" event={"ID":"8876af21-3057-4be2-8ea4-ae681b457ccc","Type":"ContainerDied","Data":"0a7c020edefca8053bc7ec4dbe31477d1d2ae730390f21c2f5d3421c6ad056ca"} Sep 10 04:59:54.210425 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.210061 2572 scope.go:117] "RemoveContainer" containerID="c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea" Sep 10 04:59:54.211437 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.211415 2572 generic.go:356] "Generic (PLEG): container finished" podID="f2540175-2fd9-4221-a446-9b965d665c0f" containerID="5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c" exitCode=0 Sep 10 04:59:54.211534 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.211472 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" event={"ID":"f2540175-2fd9-4221-a446-9b965d665c0f","Type":"ContainerDied","Data":"5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c"} Sep 10 04:59:54.211534 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.211482 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" Sep 10 04:59:54.211534 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.211510 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx" event={"ID":"f2540175-2fd9-4221-a446-9b965d665c0f","Type":"ContainerDied","Data":"52129b4f7c3bada9a705bfe372a3940923e7d878b5b1ef355953a90d980d5604"} Sep 10 04:59:54.221064 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.221044 2572 scope.go:117] "RemoveContainer" containerID="c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea" Sep 10 04:59:54.221410 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:59:54.221390 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea\": container with ID starting with c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea not found: ID does not exist" containerID="c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea" Sep 10 04:59:54.221498 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.221425 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea"} err="failed to get container status \"c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea\": rpc error: code = NotFound desc = could not find container \"c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea\": container with ID starting with c55eda15c21ea6592e653655c098adf9b00ed67f4d4b560c5be612dea00000ea not found: ID does not exist" Sep 10 04:59:54.221498 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.221465 2572 scope.go:117] "RemoveContainer" containerID="5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c" Sep 10 04:59:54.232585 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.232559 2572 scope.go:117] "RemoveContainer" containerID="5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c" Sep 10 04:59:54.232915 ip-10-0-136-213 kubenswrapper[2572]: E0910 04:59:54.232892 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c\": container with ID starting with 5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c not found: ID does not exist" containerID="5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c" Sep 10 04:59:54.232992 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.232920 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c"} err="failed to get container status \"5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c\": rpc error: code = NotFound desc = could not find container \"5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c\": container with ID starting with 5aeb299e86839d21af3c9ba9c59fb092ac3897695bd565830c408dee5601163c not found: ID does not exist" Sep 10 04:59:54.244119 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.244087 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx"] Sep 10 04:59:54.246498 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.246471 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aa81ae79-cbfdd557-j2sjx"] Sep 10 04:59:54.259634 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.259600 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx"] Sep 10 04:59:54.262886 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:54.262853 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aa81ae79-69f5f7d6d6-bw8zx"] Sep 10 04:59:55.037814 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:55.037782 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8876af21-3057-4be2-8ea4-ae681b457ccc" path="/var/lib/kubelet/pods/8876af21-3057-4be2-8ea4-ae681b457ccc/volumes" Sep 10 04:59:55.038177 ip-10-0-136-213 kubenswrapper[2572]: I0910 04:59:55.038162 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2540175-2fd9-4221-a446-9b965d665c0f" path="/var/lib/kubelet/pods/f2540175-2fd9-4221-a446-9b965d665c0f/volumes" Sep 10 05:00:09.302158 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:09.302121 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h"] Sep 10 05:00:17.317191 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.317146 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn"] Sep 10 05:00:17.317914 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.317890 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2540175-2fd9-4221-a446-9b965d665c0f" containerName="payload-processing" Sep 10 05:00:17.317960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.317919 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2540175-2fd9-4221-a446-9b965d665c0f" containerName="payload-processing" Sep 10 05:00:17.317960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.317945 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8876af21-3057-4be2-8ea4-ae681b457ccc" containerName="payload-pre-processing" Sep 10 05:00:17.317960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.317954 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8876af21-3057-4be2-8ea4-ae681b457ccc" containerName="payload-pre-processing" Sep 10 05:00:17.318071 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.318059 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8876af21-3057-4be2-8ea4-ae681b457ccc" containerName="payload-pre-processing" Sep 10 05:00:17.318117 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.318079 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2540175-2fd9-4221-a446-9b965d665c0f" containerName="payload-processing" Sep 10 05:00:17.323556 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.323522 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.326480 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.326340 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-25bd47-serving-cert\"" Sep 10 05:00:17.329768 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.329739 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn"] Sep 10 05:00:17.473099 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.473058 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbxqg\" (UniqueName: \"kubernetes.io/projected/a279e799-9e46-41a2-8063-d7980773f91a-kube-api-access-nbxqg\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.473316 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.473161 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.473316 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.473217 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.473316 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.473281 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a279e799-9e46-41a2-8063-d7980773f91a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.574870 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.574749 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.574870 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.574816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a279e799-9e46-41a2-8063-d7980773f91a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.575115 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.574915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbxqg\" (UniqueName: \"kubernetes.io/projected/a279e799-9e46-41a2-8063-d7980773f91a-kube-api-access-nbxqg\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.575115 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.575012 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.575892 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.575865 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a279e799-9e46-41a2-8063-d7980773f91a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.577610 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.577587 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.577855 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.577831 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.583354 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.583322 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbxqg\" (UniqueName: \"kubernetes.io/projected/a279e799-9e46-41a2-8063-d7980773f91a-kube-api-access-nbxqg\") pod \"maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.637275 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.637234 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:17.915109 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.915028 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn"] Sep 10 05:00:17.918578 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.918548 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq"] Sep 10 05:00:17.919054 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:00:17.919027 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda279e799_9e46_41a2_8063_d7980773f91a.slice/crio-0d279a942ceff852c708d5a0d33f4b8bfa03cb42167d11ffd56ee1b2b42318ff WatchSource:0}: Error finding container 0d279a942ceff852c708d5a0d33f4b8bfa03cb42167d11ffd56ee1b2b42318ff: Status 404 returned error can't find the container with id 0d279a942ceff852c708d5a0d33f4b8bfa03cb42167d11ffd56ee1b2b42318ff Sep 10 05:00:17.925436 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.925411 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:00:17.927970 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.927942 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-25bd47\"" Sep 10 05:00:17.930726 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.930677 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq"] Sep 10 05:00:17.954288 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.954247 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw"] Sep 10 05:00:17.958161 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.958135 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:00:17.963682 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.963643 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-25bd47-dockercfg-wj8dr\"" Sep 10 05:00:17.967634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:17.967604 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw"] Sep 10 05:00:18.080023 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.079909 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4tq\" (UniqueName: \"kubernetes.io/projected/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-kube-api-access-mk4tq\") pod \"payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq\" (UID: \"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:00:18.080241 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.080063 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq\" (UID: \"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:00:18.080241 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.080128 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm27s\" (UniqueName: \"kubernetes.io/projected/f27cc422-1f3d-486c-b9a8-3f85495223f2-kube-api-access-xm27s\") pod \"payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw\" (UID: \"f27cc422-1f3d-486c-b9a8-3f85495223f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:00:18.080241 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.080193 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27cc422-1f3d-486c-b9a8-3f85495223f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw\" (UID: \"f27cc422-1f3d-486c-b9a8-3f85495223f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:00:18.180925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.180813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq\" (UID: \"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:00:18.180925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.180902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm27s\" (UniqueName: \"kubernetes.io/projected/f27cc422-1f3d-486c-b9a8-3f85495223f2-kube-api-access-xm27s\") pod \"payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw\" (UID: \"f27cc422-1f3d-486c-b9a8-3f85495223f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:00:18.181168 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.180968 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27cc422-1f3d-486c-b9a8-3f85495223f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw\" (UID: \"f27cc422-1f3d-486c-b9a8-3f85495223f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:00:18.181168 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.181010 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4tq\" (UniqueName: \"kubernetes.io/projected/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-kube-api-access-mk4tq\") pod \"payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq\" (UID: \"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:00:18.181809 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.181785 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq\" (UID: \"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:00:18.181893 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.181875 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27cc422-1f3d-486c-b9a8-3f85495223f2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw\" (UID: \"f27cc422-1f3d-486c-b9a8-3f85495223f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:00:18.192429 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.192398 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm27s\" (UniqueName: \"kubernetes.io/projected/f27cc422-1f3d-486c-b9a8-3f85495223f2-kube-api-access-xm27s\") pod \"payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw\" (UID: \"f27cc422-1f3d-486c-b9a8-3f85495223f2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:00:18.195800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.195776 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4tq\" (UniqueName: \"kubernetes.io/projected/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-kube-api-access-mk4tq\") pod \"payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq\" (UID: \"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:00:18.242003 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.241964 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:00:18.270673 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.270636 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:00:18.352046 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.351998 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" event={"ID":"a279e799-9e46-41a2-8063-d7980773f91a","Type":"ContainerStarted","Data":"0d279a942ceff852c708d5a0d33f4b8bfa03cb42167d11ffd56ee1b2b42318ff"} Sep 10 05:00:18.519158 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.519125 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq"] Sep 10 05:00:18.520335 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:00:18.520309 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc2f1ba_0d36_4d7b_aec8_156383da5f7b.slice/crio-f82ca10f44d4bd9304c3b90e963bef66ede81c612038d7f134e064b8a738bd54 WatchSource:0}: Error finding container f82ca10f44d4bd9304c3b90e963bef66ede81c612038d7f134e064b8a738bd54: Status 404 returned error can't find the container with id f82ca10f44d4bd9304c3b90e963bef66ede81c612038d7f134e064b8a738bd54 Sep 10 05:00:18.557899 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:18.557868 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw"] Sep 10 05:00:18.559160 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:00:18.559128 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27cc422_1f3d_486c_b9a8_3f85495223f2.slice/crio-de35e9bf646cd72032c4644503e787e832d236aeb26b8be51a1827c067939839 WatchSource:0}: Error finding container de35e9bf646cd72032c4644503e787e832d236aeb26b8be51a1827c067939839: Status 404 returned error can't find the container with id de35e9bf646cd72032c4644503e787e832d236aeb26b8be51a1827c067939839 Sep 10 05:00:19.359610 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:19.359572 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" event={"ID":"f27cc422-1f3d-486c-b9a8-3f85495223f2","Type":"ContainerStarted","Data":"c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a"} Sep 10 05:00:19.360058 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:19.359636 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:00:19.360058 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:19.359653 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" event={"ID":"f27cc422-1f3d-486c-b9a8-3f85495223f2","Type":"ContainerStarted","Data":"de35e9bf646cd72032c4644503e787e832d236aeb26b8be51a1827c067939839"} Sep 10 05:00:19.361317 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:19.361289 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" event={"ID":"a279e799-9e46-41a2-8063-d7980773f91a","Type":"ContainerStarted","Data":"a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8"} Sep 10 05:00:19.361467 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:19.361339 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:19.363063 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:19.363039 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" event={"ID":"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b","Type":"ContainerStarted","Data":"4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4"} Sep 10 05:00:19.363168 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:19.363086 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" event={"ID":"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b","Type":"ContainerStarted","Data":"f82ca10f44d4bd9304c3b90e963bef66ede81c612038d7f134e064b8a738bd54"} Sep 10 05:00:19.363168 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:19.363130 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:00:19.376741 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:19.376687 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" podStartSLOduration=2.376668232 podStartE2EDuration="2.376668232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:00:19.37484644 +0000 UTC m=+1138.878679502" watchObservedRunningTime="2026-09-10 05:00:19.376668232 +0000 UTC m=+1138.880501292" Sep 10 05:00:19.391824 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:19.391758 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" podStartSLOduration=2.391737349 podStartE2EDuration="2.391737349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:00:19.388994632 +0000 UTC m=+1138.892827693" watchObservedRunningTime="2026-09-10 05:00:19.391737349 +0000 UTC m=+1138.895570412" Sep 10 05:00:24.326255 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.326192 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" podUID="6bea8e93-1a89-4b3c-9652-77ec507783cc" containerName="main" containerID="cri-o://2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7" gracePeriod=45 Sep 10 05:00:24.584290 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.584209 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 05:00:24.604885 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.604817 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" podStartSLOduration=7.057300161 podStartE2EDuration="7.604793835s" totalImagesPullingTime="547.493674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:00:19.408677864 +0000 UTC m=+1138.912510924" watchObservedRunningTime="2026-09-10 05:00:24.604793835 +0000 UTC m=+1144.108626899" Sep 10 05:00:24.654265 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654222 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-tmp-dir\") pod \"6bea8e93-1a89-4b3c-9652-77ec507783cc\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " Sep 10 05:00:24.654504 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654302 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-home\") pod \"6bea8e93-1a89-4b3c-9652-77ec507783cc\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " Sep 10 05:00:24.654504 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654328 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-dshm\") pod \"6bea8e93-1a89-4b3c-9652-77ec507783cc\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " Sep 10 05:00:24.654504 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654378 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-model-cache\") pod \"6bea8e93-1a89-4b3c-9652-77ec507783cc\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " Sep 10 05:00:24.654504 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654410 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7ffr\" (UniqueName: \"kubernetes.io/projected/6bea8e93-1a89-4b3c-9652-77ec507783cc-kube-api-access-p7ffr\") pod \"6bea8e93-1a89-4b3c-9652-77ec507783cc\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " Sep 10 05:00:24.654504 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654435 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bea8e93-1a89-4b3c-9652-77ec507783cc-tls-certs\") pod \"6bea8e93-1a89-4b3c-9652-77ec507783cc\" (UID: \"6bea8e93-1a89-4b3c-9652-77ec507783cc\") " Sep 10 05:00:24.654761 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654523 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6bea8e93-1a89-4b3c-9652-77ec507783cc" (UID: "6bea8e93-1a89-4b3c-9652-77ec507783cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:00:24.654761 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654630 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-home" (OuterVolumeSpecName: "home") pod "6bea8e93-1a89-4b3c-9652-77ec507783cc" (UID: "6bea8e93-1a89-4b3c-9652-77ec507783cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:00:24.654761 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654676 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bea8e93-1a89-4b3c-9652-77ec507783cc" (UID: "6bea8e93-1a89-4b3c-9652-77ec507783cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:00:24.654918 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654787 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-home\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:24.654918 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654812 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-model-cache\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:24.654918 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.654828 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-tmp-dir\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:24.656765 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.656739 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bea8e93-1a89-4b3c-9652-77ec507783cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bea8e93-1a89-4b3c-9652-77ec507783cc" (UID: "6bea8e93-1a89-4b3c-9652-77ec507783cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:00:24.656765 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.656754 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-dshm" (OuterVolumeSpecName: "dshm") pod "6bea8e93-1a89-4b3c-9652-77ec507783cc" (UID: "6bea8e93-1a89-4b3c-9652-77ec507783cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:00:24.656909 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.656866 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bea8e93-1a89-4b3c-9652-77ec507783cc-kube-api-access-p7ffr" (OuterVolumeSpecName: "kube-api-access-p7ffr") pod "6bea8e93-1a89-4b3c-9652-77ec507783cc" (UID: "6bea8e93-1a89-4b3c-9652-77ec507783cc"). InnerVolumeSpecName "kube-api-access-p7ffr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:00:24.755431 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.755390 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bea8e93-1a89-4b3c-9652-77ec507783cc-dshm\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:24.755431 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.755423 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7ffr\" (UniqueName: \"kubernetes.io/projected/6bea8e93-1a89-4b3c-9652-77ec507783cc-kube-api-access-p7ffr\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:24.755431 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:24.755435 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bea8e93-1a89-4b3c-9652-77ec507783cc-tls-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:25.374997 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:25.374898 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:00:25.395647 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:25.395613 2572 generic.go:356] "Generic (PLEG): container finished" podID="6bea8e93-1a89-4b3c-9652-77ec507783cc" containerID="2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7" exitCode=0 Sep 10 05:00:25.395825 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:25.395669 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" Sep 10 05:00:25.395825 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:25.395693 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" event={"ID":"6bea8e93-1a89-4b3c-9652-77ec507783cc","Type":"ContainerDied","Data":"2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7"} Sep 10 05:00:25.395825 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:25.395738 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h" event={"ID":"6bea8e93-1a89-4b3c-9652-77ec507783cc","Type":"ContainerDied","Data":"d3321e25f1664eec098f5b0fbb3043a8ea7f7870eb4326cd06784ae0bcfc3288"} Sep 10 05:00:25.395825 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:25.395762 2572 scope.go:117] "RemoveContainer" containerID="2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7" Sep 10 05:00:25.406371 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:25.406340 2572 scope.go:117] "RemoveContainer" containerID="2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7" Sep 10 05:00:25.406879 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:00:25.406848 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7\": container with ID starting with 2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7 not found: ID does not exist" containerID="2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7" Sep 10 05:00:25.407036 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:25.407010 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7"} err="failed to get container status \"2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7\": rpc error: code = NotFound desc = could not find container \"2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7\": container with ID starting with 2f98f3b229b38f5ded176643e9db50cb057c1c4142b9a5e8e605d0933029dcc7 not found: ID does not exist" Sep 10 05:00:25.416027 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:25.415977 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h"] Sep 10 05:00:25.419124 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:25.419097 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f3cc69/test-model-f3cc69-kserve-85684fbd85-vwm2h"] Sep 10 05:00:27.037097 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:27.037064 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bea8e93-1a89-4b3c-9652-77ec507783cc" path="/var/lib/kubelet/pods/6bea8e93-1a89-4b3c-9652-77ec507783cc/volumes" Sep 10 05:00:30.371361 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:30.371318 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:00:30.371807 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:30.371403 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:00:42.397822 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:42.397776 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b"] Sep 10 05:00:42.398276 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:42.398156 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" podUID="a170ab36-72e8-4563-800e-57860915f26b" containerName="istio-proxy" containerID="cri-o://a36c8f839bab64e0bc3893f092b159aa1f8b20ec9ad852bcc2982347ed605235" gracePeriod=30 Sep 10 05:00:46.159696 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:46.159601 2572 patch_prober.go:28] interesting pod/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 05:00:46.159696 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:46.159659 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" podUID="a170ab36-72e8-4563-800e-57860915f26b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 05:00:47.154172 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.154129 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct"] Sep 10 05:00:47.154731 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.154710 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bea8e93-1a89-4b3c-9652-77ec507783cc" containerName="main" Sep 10 05:00:47.154731 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.154733 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bea8e93-1a89-4b3c-9652-77ec507783cc" containerName="main" Sep 10 05:00:47.154915 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.154860 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bea8e93-1a89-4b3c-9652-77ec507783cc" containerName="main" Sep 10 05:00:47.159881 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.159837 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:00:47.162349 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.162325 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-97ffde\"" Sep 10 05:00:47.162548 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.162527 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-97ffde-dockercfg-dj6x4\"" Sep 10 05:00:47.166693 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.166663 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct"] Sep 10 05:00:47.290495 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.288805 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd273e3b-9056-4f71-b567-9aba987d9788-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct\" (UID: \"cd273e3b-9056-4f71-b567-9aba987d9788\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:00:47.290495 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.288941 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6z4f\" (UniqueName: \"kubernetes.io/projected/cd273e3b-9056-4f71-b567-9aba987d9788-kube-api-access-c6z4f\") pod \"payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct\" (UID: \"cd273e3b-9056-4f71-b567-9aba987d9788\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:00:47.390375 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.390321 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd273e3b-9056-4f71-b567-9aba987d9788-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct\" (UID: \"cd273e3b-9056-4f71-b567-9aba987d9788\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:00:47.390554 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.390468 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6z4f\" (UniqueName: \"kubernetes.io/projected/cd273e3b-9056-4f71-b567-9aba987d9788-kube-api-access-c6z4f\") pod \"payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct\" (UID: \"cd273e3b-9056-4f71-b567-9aba987d9788\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:00:47.391115 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.391095 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd273e3b-9056-4f71-b567-9aba987d9788-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct\" (UID: \"cd273e3b-9056-4f71-b567-9aba987d9788\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:00:47.398605 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.398561 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6z4f\" (UniqueName: \"kubernetes.io/projected/cd273e3b-9056-4f71-b567-9aba987d9788-kube-api-access-c6z4f\") pod \"payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct\" (UID: \"cd273e3b-9056-4f71-b567-9aba987d9788\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:00:47.473592 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.473553 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:00:47.543476 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.543293 2572 generic.go:356] "Generic (PLEG): container finished" podID="a170ab36-72e8-4563-800e-57860915f26b" containerID="a36c8f839bab64e0bc3893f092b159aa1f8b20ec9ad852bcc2982347ed605235" exitCode=0 Sep 10 05:00:47.543476 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.543415 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" event={"ID":"a170ab36-72e8-4563-800e-57860915f26b","Type":"ContainerDied","Data":"a36c8f839bab64e0bc3893f092b159aa1f8b20ec9ad852bcc2982347ed605235"} Sep 10 05:00:47.693832 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.693796 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 05:00:47.754349 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.754302 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct"] Sep 10 05:00:47.756262 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:00:47.756223 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd273e3b_9056_4f71_b567_9aba987d9788.slice/crio-9eb56c4ff009c0cc65a07d7497530904bdf551304e2369983acdc5fc0f11d38f WatchSource:0}: Error finding container 9eb56c4ff009c0cc65a07d7497530904bdf551304e2369983acdc5fc0f11d38f: Status 404 returned error can't find the container with id 9eb56c4ff009c0cc65a07d7497530904bdf551304e2369983acdc5fc0f11d38f Sep 10 05:00:47.794619 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.794580 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cdlz\" (UniqueName: \"kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-kube-api-access-4cdlz\") pod \"a170ab36-72e8-4563-800e-57860915f26b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " Sep 10 05:00:47.794836 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.794658 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a170ab36-72e8-4563-800e-57860915f26b-istiod-ca-cert\") pod \"a170ab36-72e8-4563-800e-57860915f26b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " Sep 10 05:00:47.794836 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.794717 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-certs\") pod \"a170ab36-72e8-4563-800e-57860915f26b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " Sep 10 05:00:47.794836 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.794741 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-socket\") pod \"a170ab36-72e8-4563-800e-57860915f26b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " Sep 10 05:00:47.794836 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.794802 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-envoy\") pod \"a170ab36-72e8-4563-800e-57860915f26b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " Sep 10 05:00:47.795064 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.794856 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-istio-token\") pod \"a170ab36-72e8-4563-800e-57860915f26b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " Sep 10 05:00:47.795064 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.794893 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-credential-socket\") pod \"a170ab36-72e8-4563-800e-57860915f26b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " Sep 10 05:00:47.795064 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.794949 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-data\") pod \"a170ab36-72e8-4563-800e-57860915f26b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " Sep 10 05:00:47.795064 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.794991 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a170ab36-72e8-4563-800e-57860915f26b-istio-podinfo\") pod \"a170ab36-72e8-4563-800e-57860915f26b\" (UID: \"a170ab36-72e8-4563-800e-57860915f26b\") " Sep 10 05:00:47.795267 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.795139 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a170ab36-72e8-4563-800e-57860915f26b" (UID: "a170ab36-72e8-4563-800e-57860915f26b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:00:47.795333 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.795304 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a170ab36-72e8-4563-800e-57860915f26b" (UID: "a170ab36-72e8-4563-800e-57860915f26b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:00:47.795537 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.795516 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:47.795656 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.795547 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-workload-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:47.795656 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.795514 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a170ab36-72e8-4563-800e-57860915f26b" (UID: "a170ab36-72e8-4563-800e-57860915f26b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:00:47.796639 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.796465 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a170ab36-72e8-4563-800e-57860915f26b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a170ab36-72e8-4563-800e-57860915f26b" (UID: "a170ab36-72e8-4563-800e-57860915f26b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:00:47.796966 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.796940 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-data" (OuterVolumeSpecName: "istio-data") pod "a170ab36-72e8-4563-800e-57860915f26b" (UID: "a170ab36-72e8-4563-800e-57860915f26b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:00:47.797383 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.797349 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-istio-token" (OuterVolumeSpecName: "istio-token") pod "a170ab36-72e8-4563-800e-57860915f26b" (UID: "a170ab36-72e8-4563-800e-57860915f26b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:00:47.797608 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.797582 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a170ab36-72e8-4563-800e-57860915f26b" (UID: "a170ab36-72e8-4563-800e-57860915f26b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:00:47.797787 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.797765 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a170ab36-72e8-4563-800e-57860915f26b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a170ab36-72e8-4563-800e-57860915f26b" (UID: "a170ab36-72e8-4563-800e-57860915f26b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 05:00:47.798010 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.797982 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-kube-api-access-4cdlz" (OuterVolumeSpecName: "kube-api-access-4cdlz") pod "a170ab36-72e8-4563-800e-57860915f26b" (UID: "a170ab36-72e8-4563-800e-57860915f26b"). InnerVolumeSpecName "kube-api-access-4cdlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:00:47.896962 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.896913 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a170ab36-72e8-4563-800e-57860915f26b-istiod-ca-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:47.896962 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.896955 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-envoy\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:47.896962 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.896968 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-istio-token\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:47.897220 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.896980 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-credential-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:47.897220 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.896996 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a170ab36-72e8-4563-800e-57860915f26b-istio-data\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:47.897220 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.897007 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a170ab36-72e8-4563-800e-57860915f26b-istio-podinfo\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:47.897220 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:47.897034 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cdlz\" (UniqueName: \"kubernetes.io/projected/a170ab36-72e8-4563-800e-57860915f26b-kube-api-access-4cdlz\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:00:48.550796 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:48.550748 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" event={"ID":"cd273e3b-9056-4f71-b567-9aba987d9788","Type":"ContainerStarted","Data":"c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1"} Sep 10 05:00:48.551359 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:48.550822 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:00:48.551359 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:48.550846 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" event={"ID":"cd273e3b-9056-4f71-b567-9aba987d9788","Type":"ContainerStarted","Data":"9eb56c4ff009c0cc65a07d7497530904bdf551304e2369983acdc5fc0f11d38f"} Sep 10 05:00:48.552347 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:48.552321 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" event={"ID":"a170ab36-72e8-4563-800e-57860915f26b","Type":"ContainerDied","Data":"ec1e88f001b39ddd6b9907c29d17003348f31ec53be82fa7491cac2c71c3b12c"} Sep 10 05:00:48.552347 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:48.552344 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b" Sep 10 05:00:48.552609 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:48.552381 2572 scope.go:117] "RemoveContainer" containerID="a36c8f839bab64e0bc3893f092b159aa1f8b20ec9ad852bcc2982347ed605235" Sep 10 05:00:48.567960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:48.567900 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" podStartSLOduration=1.5678818749999999 podStartE2EDuration="1.567881875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:00:48.565959537 +0000 UTC m=+1168.069792597" watchObservedRunningTime="2026-09-10 05:00:48.567881875 +0000 UTC m=+1168.071714981" Sep 10 05:00:48.584980 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:48.584911 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b"] Sep 10 05:00:48.588963 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:48.588923 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-3e5c28-openshift-default-54bd7b5c5b-nk65b"] Sep 10 05:00:49.038344 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:49.038308 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a170ab36-72e8-4563-800e-57860915f26b" path="/var/lib/kubelet/pods/a170ab36-72e8-4563-800e-57860915f26b/volumes" Sep 10 05:00:59.563045 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:00:59.563014 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:01:15.492317 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.492276 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x"] Sep 10 05:01:15.492949 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.492549 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" podUID="255a1ddc-1b5e-491d-98c0-fcfdac6233c1" containerName="maas-api" containerID="cri-o://3f1d32e0479c2d2839dfe1aeafc5a11ee4bdcfa9642dcd95ee00d26188e5d7ea" gracePeriod=30 Sep 10 05:01:15.542651 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.542614 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz"] Sep 10 05:01:15.542901 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.542852 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" podUID="6217e20c-03ca-4fa9-b9cf-457b28c35f30" containerName="payload-processing" containerID="cri-o://af147d6d5f3e11d9a4363b6b24e1b9ef10f7e3329743179ff310feb6178faa52" gracePeriod=30 Sep 10 05:01:15.706897 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.706864 2572 generic.go:356] "Generic (PLEG): container finished" podID="6217e20c-03ca-4fa9-b9cf-457b28c35f30" containerID="af147d6d5f3e11d9a4363b6b24e1b9ef10f7e3329743179ff310feb6178faa52" exitCode=0 Sep 10 05:01:15.707113 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.707055 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" event={"ID":"6217e20c-03ca-4fa9-b9cf-457b28c35f30","Type":"ContainerDied","Data":"af147d6d5f3e11d9a4363b6b24e1b9ef10f7e3329743179ff310feb6178faa52"} Sep 10 05:01:15.709610 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.709437 2572 generic.go:356] "Generic (PLEG): container finished" podID="255a1ddc-1b5e-491d-98c0-fcfdac6233c1" containerID="3f1d32e0479c2d2839dfe1aeafc5a11ee4bdcfa9642dcd95ee00d26188e5d7ea" exitCode=0 Sep 10 05:01:15.709610 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.709491 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" event={"ID":"255a1ddc-1b5e-491d-98c0-fcfdac6233c1","Type":"ContainerDied","Data":"3f1d32e0479c2d2839dfe1aeafc5a11ee4bdcfa9642dcd95ee00d26188e5d7ea"} Sep 10 05:01:15.801991 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.801559 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 05:01:15.842334 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.842309 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 05:01:15.883150 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.883109 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k26q9\" (UniqueName: \"kubernetes.io/projected/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-kube-api-access-k26q9\") pod \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " Sep 10 05:01:15.883346 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.883167 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-maas-api-tls\") pod \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " Sep 10 05:01:15.883346 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.883283 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6217e20c-03ca-4fa9-b9cf-457b28c35f30-plugins-config-volume\") pod \"6217e20c-03ca-4fa9-b9cf-457b28c35f30\" (UID: \"6217e20c-03ca-4fa9-b9cf-457b28c35f30\") " Sep 10 05:01:15.883346 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.883329 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l72ww\" (UniqueName: \"kubernetes.io/projected/6217e20c-03ca-4fa9-b9cf-457b28c35f30-kube-api-access-l72ww\") pod \"6217e20c-03ca-4fa9-b9cf-457b28c35f30\" (UID: \"6217e20c-03ca-4fa9-b9cf-457b28c35f30\") " Sep 10 05:01:15.883559 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.883439 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-openshift-service-ca\") pod \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " Sep 10 05:01:15.883559 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.883523 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-metrics-certs\") pod \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\" (UID: \"255a1ddc-1b5e-491d-98c0-fcfdac6233c1\") " Sep 10 05:01:15.883803 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.883771 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "255a1ddc-1b5e-491d-98c0-fcfdac6233c1" (UID: "255a1ddc-1b5e-491d-98c0-fcfdac6233c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:01:15.883890 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.883782 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6217e20c-03ca-4fa9-b9cf-457b28c35f30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6217e20c-03ca-4fa9-b9cf-457b28c35f30" (UID: "6217e20c-03ca-4fa9-b9cf-457b28c35f30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:01:15.883890 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.883873 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-openshift-service-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:15.884013 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.883889 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6217e20c-03ca-4fa9-b9cf-457b28c35f30-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:15.885612 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.885580 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6217e20c-03ca-4fa9-b9cf-457b28c35f30-kube-api-access-l72ww" (OuterVolumeSpecName: "kube-api-access-l72ww") pod "6217e20c-03ca-4fa9-b9cf-457b28c35f30" (UID: "6217e20c-03ca-4fa9-b9cf-457b28c35f30"). InnerVolumeSpecName "kube-api-access-l72ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:01:15.885707 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.885593 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-kube-api-access-k26q9" (OuterVolumeSpecName: "kube-api-access-k26q9") pod "255a1ddc-1b5e-491d-98c0-fcfdac6233c1" (UID: "255a1ddc-1b5e-491d-98c0-fcfdac6233c1"). InnerVolumeSpecName "kube-api-access-k26q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:01:15.885707 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.885650 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "255a1ddc-1b5e-491d-98c0-fcfdac6233c1" (UID: "255a1ddc-1b5e-491d-98c0-fcfdac6233c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:01:15.885793 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.885778 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "255a1ddc-1b5e-491d-98c0-fcfdac6233c1" (UID: "255a1ddc-1b5e-491d-98c0-fcfdac6233c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:01:15.984589 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.984547 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:15.984589 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.984580 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k26q9\" (UniqueName: \"kubernetes.io/projected/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-kube-api-access-k26q9\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:15.984589 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.984590 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/255a1ddc-1b5e-491d-98c0-fcfdac6233c1-maas-api-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:15.984589 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:15.984599 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l72ww\" (UniqueName: \"kubernetes.io/projected/6217e20c-03ca-4fa9-b9cf-457b28c35f30-kube-api-access-l72ww\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:16.716631 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:16.716599 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" Sep 10 05:01:16.717140 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:16.716640 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x" event={"ID":"255a1ddc-1b5e-491d-98c0-fcfdac6233c1","Type":"ContainerDied","Data":"04a83823fd07b6f7588831aea0ecc46a9eb1fc9e3b624fb9c506bb70ce8b4137"} Sep 10 05:01:16.717140 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:16.716711 2572 scope.go:117] "RemoveContainer" containerID="3f1d32e0479c2d2839dfe1aeafc5a11ee4bdcfa9642dcd95ee00d26188e5d7ea" Sep 10 05:01:16.718252 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:16.718236 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" Sep 10 05:01:16.718341 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:16.718235 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz" event={"ID":"6217e20c-03ca-4fa9-b9cf-457b28c35f30","Type":"ContainerDied","Data":"ac8e49e45f5b9e7479fd411c6c713b715369d6415e7ec4b7403f72ae1f8d59fb"} Sep 10 05:01:16.727048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:16.727026 2572 scope.go:117] "RemoveContainer" containerID="af147d6d5f3e11d9a4363b6b24e1b9ef10f7e3329743179ff310feb6178faa52" Sep 10 05:01:16.742210 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:16.742180 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x"] Sep 10 05:01:16.744389 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:16.744364 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7f2cd3-569b5bb65d-snw7x"] Sep 10 05:01:16.753815 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:16.753789 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz"] Sep 10 05:01:16.759173 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:16.759098 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7f2cd3-7b657658d5-cw6vz"] Sep 10 05:01:17.037942 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:17.037860 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="255a1ddc-1b5e-491d-98c0-fcfdac6233c1" path="/var/lib/kubelet/pods/255a1ddc-1b5e-491d-98c0-fcfdac6233c1/volumes" Sep 10 05:01:17.038290 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:17.038275 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6217e20c-03ca-4fa9-b9cf-457b28c35f30" path="/var/lib/kubelet/pods/6217e20c-03ca-4fa9-b9cf-457b28c35f30/volumes" Sep 10 05:01:22.079658 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:22.079618 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l"] Sep 10 05:01:22.080109 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:22.079834 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" containerID="cri-o://64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98" gracePeriod=30 Sep 10 05:01:23.664823 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:23.664399 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q"] Sep 10 05:01:23.664823 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:23.664798 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" podUID="01c226b3-d201-4323-9c0d-3714ba0b672a" containerName="maas-api" containerID="cri-o://fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696" gracePeriod=30 Sep 10 05:01:23.924044 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:23.923958 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 05:01:24.065975 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.065939 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-metrics-certs\") pod \"01c226b3-d201-4323-9c0d-3714ba0b672a\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " Sep 10 05:01:24.066183 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.065989 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw9rq\" (UniqueName: \"kubernetes.io/projected/01c226b3-d201-4323-9c0d-3714ba0b672a-kube-api-access-dw9rq\") pod \"01c226b3-d201-4323-9c0d-3714ba0b672a\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " Sep 10 05:01:24.066183 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.066012 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-maas-api-tls\") pod \"01c226b3-d201-4323-9c0d-3714ba0b672a\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " Sep 10 05:01:24.066183 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.066145 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c226b3-d201-4323-9c0d-3714ba0b672a-openshift-service-ca\") pod \"01c226b3-d201-4323-9c0d-3714ba0b672a\" (UID: \"01c226b3-d201-4323-9c0d-3714ba0b672a\") " Sep 10 05:01:24.066984 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.066949 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c226b3-d201-4323-9c0d-3714ba0b672a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01c226b3-d201-4323-9c0d-3714ba0b672a" (UID: "01c226b3-d201-4323-9c0d-3714ba0b672a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:01:24.070982 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.068998 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c226b3-d201-4323-9c0d-3714ba0b672a-kube-api-access-dw9rq" (OuterVolumeSpecName: "kube-api-access-dw9rq") pod "01c226b3-d201-4323-9c0d-3714ba0b672a" (UID: "01c226b3-d201-4323-9c0d-3714ba0b672a"). InnerVolumeSpecName "kube-api-access-dw9rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:01:24.070982 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.069088 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01c226b3-d201-4323-9c0d-3714ba0b672a" (UID: "01c226b3-d201-4323-9c0d-3714ba0b672a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:01:24.071800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.071681 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "01c226b3-d201-4323-9c0d-3714ba0b672a" (UID: "01c226b3-d201-4323-9c0d-3714ba0b672a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:01:24.167641 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.167589 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c226b3-d201-4323-9c0d-3714ba0b672a-openshift-service-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:24.167641 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.167627 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:24.167641 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.167636 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9rq\" (UniqueName: \"kubernetes.io/projected/01c226b3-d201-4323-9c0d-3714ba0b672a-kube-api-access-dw9rq\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:24.167641 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.167647 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c226b3-d201-4323-9c0d-3714ba0b672a-maas-api-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:24.769960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.769918 2572 generic.go:356] "Generic (PLEG): container finished" podID="01c226b3-d201-4323-9c0d-3714ba0b672a" containerID="fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696" exitCode=0 Sep 10 05:01:24.770409 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.769966 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" Sep 10 05:01:24.770409 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.770008 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" event={"ID":"01c226b3-d201-4323-9c0d-3714ba0b672a","Type":"ContainerDied","Data":"fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696"} Sep 10 05:01:24.770409 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.770067 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q" event={"ID":"01c226b3-d201-4323-9c0d-3714ba0b672a","Type":"ContainerDied","Data":"f50ea44d2d5e078d04d55efe930da5d2705354c007e9ebcb4e4e5d28f68169ad"} Sep 10 05:01:24.770409 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.770098 2572 scope.go:117] "RemoveContainer" containerID="fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696" Sep 10 05:01:24.780539 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.780518 2572 scope.go:117] "RemoveContainer" containerID="fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696" Sep 10 05:01:24.780857 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:01:24.780836 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696\": container with ID starting with fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696 not found: ID does not exist" containerID="fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696" Sep 10 05:01:24.780932 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.780867 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696"} err="failed to get container status \"fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696\": rpc error: code = NotFound desc = could not find container \"fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696\": container with ID starting with fd8fbe5545bdf571c0da38d5ac32fd5eeaaa41b75ecdb6ae4e188059371f6696 not found: ID does not exist" Sep 10 05:01:24.794303 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.794265 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q"] Sep 10 05:01:24.796736 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.796712 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-13af27-547c5997cf-hgv5q"] Sep 10 05:01:24.840067 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.839992 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl"] Sep 10 05:01:24.840674 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840648 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="255a1ddc-1b5e-491d-98c0-fcfdac6233c1" containerName="maas-api" Sep 10 05:01:24.840674 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840674 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="255a1ddc-1b5e-491d-98c0-fcfdac6233c1" containerName="maas-api" Sep 10 05:01:24.840840 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840687 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c226b3-d201-4323-9c0d-3714ba0b672a" containerName="maas-api" Sep 10 05:01:24.840840 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840695 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c226b3-d201-4323-9c0d-3714ba0b672a" containerName="maas-api" Sep 10 05:01:24.840840 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840709 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a170ab36-72e8-4563-800e-57860915f26b" containerName="istio-proxy" Sep 10 05:01:24.840840 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840719 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a170ab36-72e8-4563-800e-57860915f26b" containerName="istio-proxy" Sep 10 05:01:24.840840 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840748 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6217e20c-03ca-4fa9-b9cf-457b28c35f30" containerName="payload-processing" Sep 10 05:01:24.840840 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840753 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6217e20c-03ca-4fa9-b9cf-457b28c35f30" containerName="payload-processing" Sep 10 05:01:24.840840 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840838 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="255a1ddc-1b5e-491d-98c0-fcfdac6233c1" containerName="maas-api" Sep 10 05:01:24.841190 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840848 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c226b3-d201-4323-9c0d-3714ba0b672a" containerName="maas-api" Sep 10 05:01:24.841190 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840858 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6217e20c-03ca-4fa9-b9cf-457b28c35f30" containerName="payload-processing" Sep 10 05:01:24.841190 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.840865 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a170ab36-72e8-4563-800e-57860915f26b" containerName="istio-proxy" Sep 10 05:01:24.845698 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.845669 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:24.848176 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.848150 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8fcf3f8b-serving-cert\"" Sep 10 05:01:24.854559 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.854086 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl"] Sep 10 05:01:24.976183 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.976143 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-metrics-certs\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:24.976406 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.976317 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-maas-api-tls\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:24.976406 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.976356 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-878vw\" (UniqueName: \"kubernetes.io/projected/199b82e3-925c-48df-b3c3-cce699d21fd7-kube-api-access-878vw\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:24.976563 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:24.976439 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/199b82e3-925c-48df-b3c3-cce699d21fd7-openshift-service-ca\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:25.038251 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.038159 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01c226b3-d201-4323-9c0d-3714ba0b672a" path="/var/lib/kubelet/pods/01c226b3-d201-4323-9c0d-3714ba0b672a/volumes" Sep 10 05:01:25.077875 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.077833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-maas-api-tls\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:25.077875 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.077874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-878vw\" (UniqueName: \"kubernetes.io/projected/199b82e3-925c-48df-b3c3-cce699d21fd7-kube-api-access-878vw\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:25.078102 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.077916 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/199b82e3-925c-48df-b3c3-cce699d21fd7-openshift-service-ca\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:25.078102 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.077979 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-metrics-certs\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:25.078660 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.078637 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/199b82e3-925c-48df-b3c3-cce699d21fd7-openshift-service-ca\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:25.080332 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.080308 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-metrics-certs\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:25.080426 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.080360 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-maas-api-tls\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:25.085805 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.085774 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-878vw\" (UniqueName: \"kubernetes.io/projected/199b82e3-925c-48df-b3c3-cce699d21fd7-kube-api-access-878vw\") pod \"maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:25.159563 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.159530 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:25.437259 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.437226 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl"] Sep 10 05:01:25.439063 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:01:25.439035 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199b82e3_925c_48df_b3c3_cce699d21fd7.slice/crio-9e284cd7f8eb0efd893cbd86509fd3879b50897efa9d4b8ff0c164225ac91b7e WatchSource:0}: Error finding container 9e284cd7f8eb0efd893cbd86509fd3879b50897efa9d4b8ff0c164225ac91b7e: Status 404 returned error can't find the container with id 9e284cd7f8eb0efd893cbd86509fd3879b50897efa9d4b8ff0c164225ac91b7e Sep 10 05:01:25.781010 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.780976 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" event={"ID":"199b82e3-925c-48df-b3c3-cce699d21fd7","Type":"ContainerStarted","Data":"9e284cd7f8eb0efd893cbd86509fd3879b50897efa9d4b8ff0c164225ac91b7e"} Sep 10 05:01:25.814958 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.814914 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7"] Sep 10 05:01:25.832612 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.832203 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7"] Sep 10 05:01:25.832612 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.832361 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:25.835149 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.835121 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-4bfef6-serving-cert\"" Sep 10 05:01:25.885796 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.885752 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:25.885979 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.885838 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8mpq\" (UniqueName: \"kubernetes.io/projected/0c931a6d-1b88-417d-856f-43a1c3526a05-kube-api-access-k8mpq\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:25.885979 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.885874 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c931a6d-1b88-417d-856f-43a1c3526a05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:25.886064 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.885969 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:25.986653 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.986615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c931a6d-1b88-417d-856f-43a1c3526a05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:25.986843 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.986685 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:25.986843 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.986736 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:25.986843 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.986794 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mpq\" (UniqueName: \"kubernetes.io/projected/0c931a6d-1b88-417d-856f-43a1c3526a05-kube-api-access-k8mpq\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:25.986977 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:01:25.986854 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-serving-cert: secret "maas-api-e2e-shared-a-w3-4bfef6-serving-cert" not found Sep 10 05:01:25.986977 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:01:25.986933 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-maas-api-tls podName:0c931a6d-1b88-417d-856f-43a1c3526a05 nodeName:}" failed. No retries permitted until 2026-09-10 05:01:26.486912236 +0000 UTC m=+1205.990745294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-maas-api-tls") pod "maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" (UID: "0c931a6d-1b88-417d-856f-43a1c3526a05") : secret "maas-api-e2e-shared-a-w3-4bfef6-serving-cert" not found Sep 10 05:01:25.987291 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.987270 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c931a6d-1b88-417d-856f-43a1c3526a05-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:25.989200 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:25.989179 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:26.002761 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:26.002727 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8mpq\" (UniqueName: \"kubernetes.io/projected/0c931a6d-1b88-417d-856f-43a1c3526a05-kube-api-access-k8mpq\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:26.492084 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:26.492040 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:26.494593 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:26.494570 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:26.745530 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:26.745496 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:26.790106 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:26.790069 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" event={"ID":"199b82e3-925c-48df-b3c3-cce699d21fd7","Type":"ContainerStarted","Data":"d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352"} Sep 10 05:01:26.790588 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:26.790212 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:26.813579 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:26.813522 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" podStartSLOduration=1.9288710249999999 podStartE2EDuration="2.813505412s" totalImagesPullingTime="884.634387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:01:26.812439011 +0000 UTC m=+1206.316272074" watchObservedRunningTime="2026-09-10 05:01:26.813505412 +0000 UTC m=+1206.317338454" Sep 10 05:01:27.025960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.025930 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7"] Sep 10 05:01:27.028373 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:01:27.028341 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c931a6d_1b88_417d_856f_43a1c3526a05.slice/crio-eb7f9b7ec826803ad49caf5154075236e8ac2b3995fd3ffed173ce6e1f25cb88 WatchSource:0}: Error finding container eb7f9b7ec826803ad49caf5154075236e8ac2b3995fd3ffed173ce6e1f25cb88: Status 404 returned error can't find the container with id eb7f9b7ec826803ad49caf5154075236e8ac2b3995fd3ffed173ce6e1f25cb88 Sep 10 05:01:27.344205 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.344177 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 05:01:27.400790 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.400677 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-token\") pod \"d84205a9-1655-4123-9b56-c1f235f11ddb\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " Sep 10 05:01:27.401567 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.401524 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-credential-socket\") pod \"d84205a9-1655-4123-9b56-c1f235f11ddb\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " Sep 10 05:01:27.401710 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.401611 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-envoy\") pod \"d84205a9-1655-4123-9b56-c1f235f11ddb\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " Sep 10 05:01:27.401710 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.401665 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-data\") pod \"d84205a9-1655-4123-9b56-c1f235f11ddb\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " Sep 10 05:01:27.401935 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.401904 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d84205a9-1655-4123-9b56-c1f235f11ddb" (UID: "d84205a9-1655-4123-9b56-c1f235f11ddb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:01:27.402001 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.401921 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84205a9-1655-4123-9b56-c1f235f11ddb-istiod-ca-cert\") pod \"d84205a9-1655-4123-9b56-c1f235f11ddb\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " Sep 10 05:01:27.403054 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.402874 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-data" (OuterVolumeSpecName: "istio-data") pod "d84205a9-1655-4123-9b56-c1f235f11ddb" (UID: "d84205a9-1655-4123-9b56-c1f235f11ddb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:01:27.403054 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.402944 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-podinfo\") pod \"d84205a9-1655-4123-9b56-c1f235f11ddb\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " Sep 10 05:01:27.403973 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.403343 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-socket\") pod \"d84205a9-1655-4123-9b56-c1f235f11ddb\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " Sep 10 05:01:27.403973 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.403404 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-certs\") pod \"d84205a9-1655-4123-9b56-c1f235f11ddb\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " Sep 10 05:01:27.403973 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.403470 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsfhj\" (UniqueName: \"kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-kube-api-access-tsfhj\") pod \"d84205a9-1655-4123-9b56-c1f235f11ddb\" (UID: \"d84205a9-1655-4123-9b56-c1f235f11ddb\") " Sep 10 05:01:27.403973 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.403925 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-credential-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:27.403973 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.403943 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-data\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:27.404323 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.404284 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d84205a9-1655-4123-9b56-c1f235f11ddb" (UID: "d84205a9-1655-4123-9b56-c1f235f11ddb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:01:27.404404 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.404357 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d84205a9-1655-4123-9b56-c1f235f11ddb" (UID: "d84205a9-1655-4123-9b56-c1f235f11ddb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:01:27.404404 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.404362 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-token" (OuterVolumeSpecName: "istio-token") pod "d84205a9-1655-4123-9b56-c1f235f11ddb" (UID: "d84205a9-1655-4123-9b56-c1f235f11ddb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:01:27.404596 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.404567 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84205a9-1655-4123-9b56-c1f235f11ddb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d84205a9-1655-4123-9b56-c1f235f11ddb" (UID: "d84205a9-1655-4123-9b56-c1f235f11ddb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:01:27.406052 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.406021 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d84205a9-1655-4123-9b56-c1f235f11ddb" (UID: "d84205a9-1655-4123-9b56-c1f235f11ddb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:01:27.406902 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.406859 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d84205a9-1655-4123-9b56-c1f235f11ddb" (UID: "d84205a9-1655-4123-9b56-c1f235f11ddb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 05:01:27.407137 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.407101 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-kube-api-access-tsfhj" (OuterVolumeSpecName: "kube-api-access-tsfhj") pod "d84205a9-1655-4123-9b56-c1f235f11ddb" (UID: "d84205a9-1655-4123-9b56-c1f235f11ddb"). InnerVolumeSpecName "kube-api-access-tsfhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:01:27.504555 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.504517 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:27.504555 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.504552 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsfhj\" (UniqueName: \"kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-kube-api-access-tsfhj\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:27.504555 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.504565 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-token\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:27.504790 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.504574 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-envoy\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:27.504790 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.504582 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d84205a9-1655-4123-9b56-c1f235f11ddb-istiod-ca-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:27.504790 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.504590 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d84205a9-1655-4123-9b56-c1f235f11ddb-istio-podinfo\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:27.504790 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.504599 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d84205a9-1655-4123-9b56-c1f235f11ddb-workload-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:27.797840 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.797807 2572 generic.go:356] "Generic (PLEG): container finished" podID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerID="64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98" exitCode=0 Sep 10 05:01:27.798322 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.797860 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" Sep 10 05:01:27.798322 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.797898 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" event={"ID":"d84205a9-1655-4123-9b56-c1f235f11ddb","Type":"ContainerDied","Data":"64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98"} Sep 10 05:01:27.798322 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.797963 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l" event={"ID":"d84205a9-1655-4123-9b56-c1f235f11ddb","Type":"ContainerDied","Data":"db5a950bc7307c9293516a5df6c233fb5aa0f011df4a9e6d104904f1e6f4acbd"} Sep 10 05:01:27.798322 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.797999 2572 scope.go:117] "RemoveContainer" containerID="64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98" Sep 10 05:01:27.799781 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.799753 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" event={"ID":"0c931a6d-1b88-417d-856f-43a1c3526a05","Type":"ContainerStarted","Data":"e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9"} Sep 10 05:01:27.799905 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.799798 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" event={"ID":"0c931a6d-1b88-417d-856f-43a1c3526a05","Type":"ContainerStarted","Data":"eb7f9b7ec826803ad49caf5154075236e8ac2b3995fd3ffed173ce6e1f25cb88"} Sep 10 05:01:27.807471 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.807428 2572 scope.go:117] "RemoveContainer" containerID="ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0" Sep 10 05:01:27.816624 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.816596 2572 scope.go:117] "RemoveContainer" containerID="64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98" Sep 10 05:01:27.816943 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:01:27.816923 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98\": container with ID starting with 64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98 not found: ID does not exist" containerID="64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98" Sep 10 05:01:27.816991 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.816954 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98"} err="failed to get container status \"64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98\": rpc error: code = NotFound desc = could not find container \"64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98\": container with ID starting with 64ea7542f2ca2869eb7351b8797cb2ba8f04cb4c34a85ba8ab875072f8ae1f98 not found: ID does not exist" Sep 10 05:01:27.816991 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.816975 2572 scope.go:117] "RemoveContainer" containerID="ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0" Sep 10 05:01:27.817271 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:01:27.817250 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0\": container with ID starting with ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0 not found: ID does not exist" containerID="ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0" Sep 10 05:01:27.817337 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.817275 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0"} err="failed to get container status \"ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0\": rpc error: code = NotFound desc = could not find container \"ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0\": container with ID starting with ac8cb538af54e7eb239507eb7d8028d6b8b85dfd36098a4c8414536395d368c0 not found: ID does not exist" Sep 10 05:01:27.822143 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.822072 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" podStartSLOduration=2.432918549 podStartE2EDuration="2.822049362s" totalImagesPullingTime="389.130813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:01:27.821560967 +0000 UTC m=+1207.325394031" watchObservedRunningTime="2026-09-10 05:01:27.822049362 +0000 UTC m=+1207.325882424" Sep 10 05:01:27.838847 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.838807 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l"] Sep 10 05:01:27.844160 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:27.844130 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7f2cd3-openshift-default-5db5ff7c74-clb9l"] Sep 10 05:01:28.807138 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:28.807096 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:29.037752 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:29.037717 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" path="/var/lib/kubelet/pods/d84205a9-1655-4123-9b56-c1f235f11ddb/volumes" Sep 10 05:01:30.131294 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:30.131252 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5"] Sep 10 05:01:30.131744 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:30.131509 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" containerID="cri-o://cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8" gracePeriod=30 Sep 10 05:01:32.805606 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:32.805570 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:01:33.496207 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:33.496174 2572 patch_prober.go:28] interesting pod/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 05:01:33.496376 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:33.496243 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 05:01:34.820299 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:34.820267 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:01:35.395150 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.395073 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 05:01:35.486125 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486077 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-credential-socket\") pod \"8da54c73-ad57-4436-8b1f-0f421816feff\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " Sep 10 05:01:35.486125 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486129 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-socket\") pod \"8da54c73-ad57-4436-8b1f-0f421816feff\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " Sep 10 05:01:35.486393 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486154 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da54c73-ad57-4436-8b1f-0f421816feff-istiod-ca-cert\") pod \"8da54c73-ad57-4436-8b1f-0f421816feff\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " Sep 10 05:01:35.486393 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486190 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-data\") pod \"8da54c73-ad57-4436-8b1f-0f421816feff\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " Sep 10 05:01:35.486393 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486216 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-certs\") pod \"8da54c73-ad57-4436-8b1f-0f421816feff\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " Sep 10 05:01:35.486393 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486303 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-envoy\") pod \"8da54c73-ad57-4436-8b1f-0f421816feff\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " Sep 10 05:01:35.486393 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486332 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-istio-token\") pod \"8da54c73-ad57-4436-8b1f-0f421816feff\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " Sep 10 05:01:35.486660 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486406 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da54c73-ad57-4436-8b1f-0f421816feff-istio-podinfo\") pod \"8da54c73-ad57-4436-8b1f-0f421816feff\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " Sep 10 05:01:35.486660 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486435 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgff4\" (UniqueName: \"kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-kube-api-access-dgff4\") pod \"8da54c73-ad57-4436-8b1f-0f421816feff\" (UID: \"8da54c73-ad57-4436-8b1f-0f421816feff\") " Sep 10 05:01:35.486660 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486499 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8da54c73-ad57-4436-8b1f-0f421816feff" (UID: "8da54c73-ad57-4436-8b1f-0f421816feff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:01:35.486660 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486520 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8da54c73-ad57-4436-8b1f-0f421816feff" (UID: "8da54c73-ad57-4436-8b1f-0f421816feff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:01:35.486660 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486612 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-data" (OuterVolumeSpecName: "istio-data") pod "8da54c73-ad57-4436-8b1f-0f421816feff" (UID: "8da54c73-ad57-4436-8b1f-0f421816feff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:01:35.486874 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486676 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da54c73-ad57-4436-8b1f-0f421816feff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8da54c73-ad57-4436-8b1f-0f421816feff" (UID: "8da54c73-ad57-4436-8b1f-0f421816feff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:01:35.486915 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486895 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8da54c73-ad57-4436-8b1f-0f421816feff" (UID: "8da54c73-ad57-4436-8b1f-0f421816feff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:01:35.486971 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486956 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da54c73-ad57-4436-8b1f-0f421816feff-istiod-ca-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:35.487020 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486976 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-data\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:35.487020 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486985 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-credential-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:35.487020 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.486995 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:35.488887 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.488855 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-kube-api-access-dgff4" (OuterVolumeSpecName: "kube-api-access-dgff4") pod "8da54c73-ad57-4436-8b1f-0f421816feff" (UID: "8da54c73-ad57-4436-8b1f-0f421816feff"). InnerVolumeSpecName "kube-api-access-dgff4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:01:35.488887 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.488867 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8da54c73-ad57-4436-8b1f-0f421816feff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8da54c73-ad57-4436-8b1f-0f421816feff" (UID: "8da54c73-ad57-4436-8b1f-0f421816feff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 05:01:35.488887 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.488880 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-istio-token" (OuterVolumeSpecName: "istio-token") pod "8da54c73-ad57-4436-8b1f-0f421816feff" (UID: "8da54c73-ad57-4436-8b1f-0f421816feff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:01:35.489084 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.488861 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8da54c73-ad57-4436-8b1f-0f421816feff" (UID: "8da54c73-ad57-4436-8b1f-0f421816feff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:01:35.587791 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.587749 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-workload-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:35.587791 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.587783 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da54c73-ad57-4436-8b1f-0f421816feff-istio-envoy\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:35.587791 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.587794 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-istio-token\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:35.588041 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.587805 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da54c73-ad57-4436-8b1f-0f421816feff-istio-podinfo\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:35.588041 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.587816 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgff4\" (UniqueName: \"kubernetes.io/projected/8da54c73-ad57-4436-8b1f-0f421816feff-kube-api-access-dgff4\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:01:35.848956 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.848918 2572 generic.go:356] "Generic (PLEG): container finished" podID="8da54c73-ad57-4436-8b1f-0f421816feff" containerID="cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8" exitCode=0 Sep 10 05:01:35.849437 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.848964 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" event={"ID":"8da54c73-ad57-4436-8b1f-0f421816feff","Type":"ContainerDied","Data":"cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8"} Sep 10 05:01:35.849437 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.848969 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" Sep 10 05:01:35.849437 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.849011 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5" event={"ID":"8da54c73-ad57-4436-8b1f-0f421816feff","Type":"ContainerDied","Data":"35043f3a14b60b3dbeabd8687d51de3c707bd68ebd92cf8ce6254aa889291967"} Sep 10 05:01:35.849437 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.849035 2572 scope.go:117] "RemoveContainer" containerID="cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8" Sep 10 05:01:35.859056 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.859026 2572 scope.go:117] "RemoveContainer" containerID="1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4" Sep 10 05:01:35.868036 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.867992 2572 scope.go:117] "RemoveContainer" containerID="cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8" Sep 10 05:01:35.868876 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:01:35.868845 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8\": container with ID starting with cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8 not found: ID does not exist" containerID="cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8" Sep 10 05:01:35.868992 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.868887 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8"} err="failed to get container status \"cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8\": rpc error: code = NotFound desc = could not find container \"cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8\": container with ID starting with cc8c5ce4140d6985a3d82cdd35595bbf8f49015706af778539d3d51563cb66a8 not found: ID does not exist" Sep 10 05:01:35.868992 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.868912 2572 scope.go:117] "RemoveContainer" containerID="1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4" Sep 10 05:01:35.869269 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:01:35.869250 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4\": container with ID starting with 1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4 not found: ID does not exist" containerID="1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4" Sep 10 05:01:35.869320 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.869275 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4"} err="failed to get container status \"1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4\": rpc error: code = NotFound desc = could not find container \"1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4\": container with ID starting with 1b481102ebbf6635f9e12b23cf6016e6b4d12750af388366c175530263143db4 not found: ID does not exist" Sep 10 05:01:35.871001 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.870979 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5"] Sep 10 05:01:35.874802 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:35.874772 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-13af27-openshift-default-56d6fc7588-sb2b5"] Sep 10 05:01:37.037762 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:37.037727 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" path="/var/lib/kubelet/pods/8da54c73-ad57-4436-8b1f-0f421816feff/volumes" Sep 10 05:01:49.337822 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.337780 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46"] Sep 10 05:01:49.340652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.338337 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" Sep 10 05:01:49.340652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.338358 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" Sep 10 05:01:49.340652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.338371 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" Sep 10 05:01:49.340652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.338383 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" Sep 10 05:01:49.340652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.338401 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" Sep 10 05:01:49.340652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.338410 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" Sep 10 05:01:49.340652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.338567 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" Sep 10 05:01:49.340652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.338587 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" Sep 10 05:01:49.340652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.338599 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" Sep 10 05:01:49.342611 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.342584 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.345211 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.345183 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3b702d-openshift-default-dockercfg-lkl6m\"" Sep 10 05:01:49.354260 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.354226 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46"] Sep 10 05:01:49.424106 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.424066 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-credential-socket\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.424339 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.424132 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-data\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.424339 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.424153 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-token\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.424339 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.424264 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-certs\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.424339 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.424305 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-envoy\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.424567 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.424371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29deda9f-e309-4a78-a24b-465dd5b2ce00-istiod-ca-cert\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.424567 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.424417 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-socket\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.424567 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.424488 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxp9v\" (UniqueName: \"kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-kube-api-access-jxp9v\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.424567 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.424506 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-podinfo\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.525643 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.525597 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-certs\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.525643 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.525642 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-envoy\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.525899 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.525671 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29deda9f-e309-4a78-a24b-465dd5b2ce00-istiod-ca-cert\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.525899 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.525716 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-socket\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.525899 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.525769 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxp9v\" (UniqueName: \"kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-kube-api-access-jxp9v\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.526329 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.525902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-podinfo\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.526329 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.525982 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-credential-socket\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.526329 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.526052 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-certs\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.526329 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.526082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-data\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.526329 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.526124 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-token\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.526329 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.526166 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-socket\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.526329 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.526322 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-credential-socket\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.526651 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.526517 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-data\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.526651 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.526540 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29deda9f-e309-4a78-a24b-465dd5b2ce00-istiod-ca-cert\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.528027 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.528007 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-envoy\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.528422 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.528402 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-podinfo\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.533859 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.533823 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-token\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.534015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.533909 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxp9v\" (UniqueName: \"kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-kube-api-access-jxp9v\") pod \"e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.657169 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.657074 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:49.942264 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.942231 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46"] Sep 10 05:01:49.944602 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:01:49.944572 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29deda9f_e309_4a78_a24b_465dd5b2ce00.slice/crio-4110027f9e8d6a7e512dfef620915922df91ffb0278438add1f0e347d6253033 WatchSource:0}: Error finding container 4110027f9e8d6a7e512dfef620915922df91ffb0278438add1f0e347d6253033: Status 404 returned error can't find the container with id 4110027f9e8d6a7e512dfef620915922df91ffb0278438add1f0e347d6253033 Sep 10 05:01:49.947265 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.947232 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:01:49.947368 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.947317 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:01:49.947431 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:49.947361 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:01:50.930481 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:50.930433 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" event={"ID":"29deda9f-e309-4a78-a24b-465dd5b2ce00","Type":"ContainerStarted","Data":"d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40"} Sep 10 05:01:50.930880 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:50.930497 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" event={"ID":"29deda9f-e309-4a78-a24b-465dd5b2ce00","Type":"ContainerStarted","Data":"4110027f9e8d6a7e512dfef620915922df91ffb0278438add1f0e347d6253033"} Sep 10 05:01:50.951760 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:50.951703 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" podStartSLOduration=1.951683936 podStartE2EDuration="1.951683936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:01:50.950124067 +0000 UTC m=+1230.453957110" watchObservedRunningTime="2026-09-10 05:01:50.951683936 +0000 UTC m=+1230.455516996" Sep 10 05:01:51.657660 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:51.657617 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:51.663344 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:51.663313 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:51.935984 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:51.935902 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:51.936855 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:51.936838 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:01:54.636255 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.636220 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q"] Sep 10 05:01:54.636874 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.636847 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" Sep 10 05:01:54.636976 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.636875 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da54c73-ad57-4436-8b1f-0f421816feff" containerName="istio-proxy" Sep 10 05:01:54.637030 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.637012 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d84205a9-1655-4123-9b56-c1f235f11ddb" containerName="istio-proxy" Sep 10 05:01:54.640357 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.640334 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:01:54.642946 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.642919 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-3b702d\"" Sep 10 05:01:54.643075 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.643057 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-3b702d-dockercfg-kxk8g\"" Sep 10 05:01:54.650207 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.650177 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q"] Sep 10 05:01:54.677354 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.677313 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qrk\" (UniqueName: \"kubernetes.io/projected/be95fba4-fccd-40a1-ad3f-ae114d98f754-kube-api-access-n2qrk\") pod \"payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q\" (UID: \"be95fba4-fccd-40a1-ad3f-ae114d98f754\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:01:54.677572 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.677551 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be95fba4-fccd-40a1-ad3f-ae114d98f754-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q\" (UID: \"be95fba4-fccd-40a1-ad3f-ae114d98f754\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:01:54.778362 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.778320 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qrk\" (UniqueName: \"kubernetes.io/projected/be95fba4-fccd-40a1-ad3f-ae114d98f754-kube-api-access-n2qrk\") pod \"payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q\" (UID: \"be95fba4-fccd-40a1-ad3f-ae114d98f754\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:01:54.778559 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.778442 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be95fba4-fccd-40a1-ad3f-ae114d98f754-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q\" (UID: \"be95fba4-fccd-40a1-ad3f-ae114d98f754\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:01:54.779128 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.779108 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be95fba4-fccd-40a1-ad3f-ae114d98f754-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q\" (UID: \"be95fba4-fccd-40a1-ad3f-ae114d98f754\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:01:54.786330 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.786311 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qrk\" (UniqueName: \"kubernetes.io/projected/be95fba4-fccd-40a1-ad3f-ae114d98f754-kube-api-access-n2qrk\") pod \"payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q\" (UID: \"be95fba4-fccd-40a1-ad3f-ae114d98f754\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:01:54.954056 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:54.953974 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:01:55.228614 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:55.228502 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q"] Sep 10 05:01:55.231386 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:01:55.231349 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe95fba4_fccd_40a1_ad3f_ae114d98f754.slice/crio-b196dd0faab0d9593ce4e90f838d41c7bead274885cd80498fcbccf8d5a397b5 WatchSource:0}: Error finding container b196dd0faab0d9593ce4e90f838d41c7bead274885cd80498fcbccf8d5a397b5: Status 404 returned error can't find the container with id b196dd0faab0d9593ce4e90f838d41c7bead274885cd80498fcbccf8d5a397b5 Sep 10 05:01:55.958761 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:55.958720 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" event={"ID":"be95fba4-fccd-40a1-ad3f-ae114d98f754","Type":"ContainerStarted","Data":"d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077"} Sep 10 05:01:55.959171 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:55.958820 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" event={"ID":"be95fba4-fccd-40a1-ad3f-ae114d98f754","Type":"ContainerStarted","Data":"b196dd0faab0d9593ce4e90f838d41c7bead274885cd80498fcbccf8d5a397b5"} Sep 10 05:01:55.959171 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:55.958942 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:01:55.975697 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:01:55.975636 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" podStartSLOduration=1.97561886 podStartE2EDuration="1.97561886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:01:55.973905151 +0000 UTC m=+1235.477738216" watchObservedRunningTime="2026-09-10 05:01:55.97561886 +0000 UTC m=+1235.479451921" Sep 10 05:02:06.966650 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:06.966612 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:02:07.977654 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:07.977602 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 05:02:07.978157 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:07.977694 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 05:02:09.041787 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:09.041701 2572 generic.go:356] "Generic (PLEG): container finished" podID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerID="d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40" exitCode=0 Sep 10 05:02:09.041787 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:09.041762 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" event={"ID":"29deda9f-e309-4a78-a24b-465dd5b2ce00","Type":"ContainerDied","Data":"d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40"} Sep 10 05:02:09.042272 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:09.042200 2572 scope.go:117] "RemoveContainer" containerID="d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40" Sep 10 05:02:09.043352 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:09.043322 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:02:09.043443 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:09.043390 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:02:09.043443 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:09.043423 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:02:10.049646 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:10.049613 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" event={"ID":"29deda9f-e309-4a78-a24b-465dd5b2ce00","Type":"ContainerStarted","Data":"6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830"} Sep 10 05:02:10.657179 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:10.657142 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:02:10.657419 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:10.657394 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 10 05:02:10.657496 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:10.657475 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 10 05:02:11.658078 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:11.658039 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 10 05:02:11.658522 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:11.658100 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 10 05:02:12.657877 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:12.657832 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 10 05:02:12.658068 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:12.657893 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 10 05:02:13.658032 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:13.657941 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 10 05:02:13.658032 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:13.658011 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 10 05:02:14.657956 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:14.657917 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 10 05:02:14.658163 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:14.657978 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 10 05:02:15.657935 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:15.657889 2572 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 10 05:02:15.658137 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:15.657964 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 10 05:02:16.661513 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:16.661477 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:02:16.661980 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:16.661894 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:02:16.662784 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:02:16.662762 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:03:09.925389 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:09.925349 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl"] Sep 10 05:03:09.925856 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:09.925641 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" podUID="199b82e3-925c-48df-b3c3-cce699d21fd7" containerName="maas-api" containerID="cri-o://d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352" gracePeriod=30 Sep 10 05:03:10.189331 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.189255 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:03:10.267165 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.267126 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-maas-api-tls\") pod \"199b82e3-925c-48df-b3c3-cce699d21fd7\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " Sep 10 05:03:10.267352 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.267198 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-metrics-certs\") pod \"199b82e3-925c-48df-b3c3-cce699d21fd7\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " Sep 10 05:03:10.267352 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.267227 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-878vw\" (UniqueName: \"kubernetes.io/projected/199b82e3-925c-48df-b3c3-cce699d21fd7-kube-api-access-878vw\") pod \"199b82e3-925c-48df-b3c3-cce699d21fd7\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " Sep 10 05:03:10.267352 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.267251 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/199b82e3-925c-48df-b3c3-cce699d21fd7-openshift-service-ca\") pod \"199b82e3-925c-48df-b3c3-cce699d21fd7\" (UID: \"199b82e3-925c-48df-b3c3-cce699d21fd7\") " Sep 10 05:03:10.267665 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.267639 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199b82e3-925c-48df-b3c3-cce699d21fd7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "199b82e3-925c-48df-b3c3-cce699d21fd7" (UID: "199b82e3-925c-48df-b3c3-cce699d21fd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:03:10.267798 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.267786 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/199b82e3-925c-48df-b3c3-cce699d21fd7-openshift-service-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:03:10.269501 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.269418 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "199b82e3-925c-48df-b3c3-cce699d21fd7" (UID: "199b82e3-925c-48df-b3c3-cce699d21fd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:03:10.269501 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.269490 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "199b82e3-925c-48df-b3c3-cce699d21fd7" (UID: "199b82e3-925c-48df-b3c3-cce699d21fd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:03:10.269664 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.269495 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199b82e3-925c-48df-b3c3-cce699d21fd7-kube-api-access-878vw" (OuterVolumeSpecName: "kube-api-access-878vw") pod "199b82e3-925c-48df-b3c3-cce699d21fd7" (UID: "199b82e3-925c-48df-b3c3-cce699d21fd7"). InnerVolumeSpecName "kube-api-access-878vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:03:10.368356 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.368316 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:03:10.368356 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.368351 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-878vw\" (UniqueName: \"kubernetes.io/projected/199b82e3-925c-48df-b3c3-cce699d21fd7-kube-api-access-878vw\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:03:10.368356 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.368361 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/199b82e3-925c-48df-b3c3-cce699d21fd7-maas-api-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:03:10.392142 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.392103 2572 generic.go:356] "Generic (PLEG): container finished" podID="199b82e3-925c-48df-b3c3-cce699d21fd7" containerID="d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352" exitCode=0 Sep 10 05:03:10.392323 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.392149 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" Sep 10 05:03:10.392323 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.392185 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" event={"ID":"199b82e3-925c-48df-b3c3-cce699d21fd7","Type":"ContainerDied","Data":"d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352"} Sep 10 05:03:10.392323 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.392236 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl" event={"ID":"199b82e3-925c-48df-b3c3-cce699d21fd7","Type":"ContainerDied","Data":"9e284cd7f8eb0efd893cbd86509fd3879b50897efa9d4b8ff0c164225ac91b7e"} Sep 10 05:03:10.392323 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.392259 2572 scope.go:117] "RemoveContainer" containerID="d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352" Sep 10 05:03:10.402390 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.402367 2572 scope.go:117] "RemoveContainer" containerID="d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352" Sep 10 05:03:10.402769 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:03:10.402750 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352\": container with ID starting with d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352 not found: ID does not exist" containerID="d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352" Sep 10 05:03:10.402827 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.402777 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352"} err="failed to get container status \"d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352\": rpc error: code = NotFound desc = could not find container \"d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352\": container with ID starting with d099c4e0a93c8b2fb8ab51051224d3c7adb0b2b57b3e65e89471dd2ca5528352 not found: ID does not exist" Sep 10 05:03:10.414828 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.414790 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl"] Sep 10 05:03:10.417787 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:10.417754 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8fcf3f8b-56747c757b-w7pcl"] Sep 10 05:03:11.037736 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:03:11.037699 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199b82e3-925c-48df-b3c3-cce699d21fd7" path="/var/lib/kubelet/pods/199b82e3-925c-48df-b3c3-cce699d21fd7/volumes" Sep 10 05:05:08.987258 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:08.987213 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg"] Sep 10 05:05:08.988002 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:08.987963 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199b82e3-925c-48df-b3c3-cce699d21fd7" containerName="maas-api" Sep 10 05:05:08.988002 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:08.987995 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="199b82e3-925c-48df-b3c3-cce699d21fd7" containerName="maas-api" Sep 10 05:05:08.988197 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:08.988116 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="199b82e3-925c-48df-b3c3-cce699d21fd7" containerName="maas-api" Sep 10 05:05:08.991722 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:08.991698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:05:08.994292 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:08.994269 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c5e779-dockercfg-ngwsv\"" Sep 10 05:05:08.994407 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:08.994270 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c5e779\"" Sep 10 05:05:09.003859 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.003828 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg"] Sep 10 05:05:09.029052 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.029007 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l"] Sep 10 05:05:09.033982 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.033950 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:05:09.044125 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.044100 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l"] Sep 10 05:05:09.177774 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.177735 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d78d427-0bc8-4271-a0b5-6c632e721ae5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l\" (UID: \"8d78d427-0bc8-4271-a0b5-6c632e721ae5\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:05:09.177774 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.177794 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4z6n\" (UniqueName: \"kubernetes.io/projected/8d78d427-0bc8-4271-a0b5-6c632e721ae5-kube-api-access-q4z6n\") pod \"payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l\" (UID: \"8d78d427-0bc8-4271-a0b5-6c632e721ae5\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:05:09.178024 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.177830 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xgp9\" (UniqueName: \"kubernetes.io/projected/1b512086-14df-402a-81e4-972c4939bfc6-kube-api-access-6xgp9\") pod \"payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg\" (UID: \"1b512086-14df-402a-81e4-972c4939bfc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:05:09.178024 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.177966 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b512086-14df-402a-81e4-972c4939bfc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg\" (UID: \"1b512086-14df-402a-81e4-972c4939bfc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:05:09.279226 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.279119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b512086-14df-402a-81e4-972c4939bfc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg\" (UID: \"1b512086-14df-402a-81e4-972c4939bfc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:05:09.279226 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.279171 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d78d427-0bc8-4271-a0b5-6c632e721ae5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l\" (UID: \"8d78d427-0bc8-4271-a0b5-6c632e721ae5\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:05:09.279226 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.279210 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4z6n\" (UniqueName: \"kubernetes.io/projected/8d78d427-0bc8-4271-a0b5-6c632e721ae5-kube-api-access-q4z6n\") pod \"payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l\" (UID: \"8d78d427-0bc8-4271-a0b5-6c632e721ae5\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:05:09.279573 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.279236 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xgp9\" (UniqueName: \"kubernetes.io/projected/1b512086-14df-402a-81e4-972c4939bfc6-kube-api-access-6xgp9\") pod \"payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg\" (UID: \"1b512086-14df-402a-81e4-972c4939bfc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:05:09.279984 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.279962 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b512086-14df-402a-81e4-972c4939bfc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg\" (UID: \"1b512086-14df-402a-81e4-972c4939bfc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:05:09.280066 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.280044 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d78d427-0bc8-4271-a0b5-6c632e721ae5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l\" (UID: \"8d78d427-0bc8-4271-a0b5-6c632e721ae5\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:05:09.287514 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.287483 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4z6n\" (UniqueName: \"kubernetes.io/projected/8d78d427-0bc8-4271-a0b5-6c632e721ae5-kube-api-access-q4z6n\") pod \"payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l\" (UID: \"8d78d427-0bc8-4271-a0b5-6c632e721ae5\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:05:09.287645 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.287600 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xgp9\" (UniqueName: \"kubernetes.io/projected/1b512086-14df-402a-81e4-972c4939bfc6-kube-api-access-6xgp9\") pod \"payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg\" (UID: \"1b512086-14df-402a-81e4-972c4939bfc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:05:09.304382 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.304342 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:05:09.347009 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.346968 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:05:09.581105 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.581078 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg"] Sep 10 05:05:09.583137 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:05:09.583106 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b512086_14df_402a_81e4_972c4939bfc6.slice/crio-456c5bd2802f60577efab7f8a04a50069a09ea6dc93231573a7180fc8cc1a6dc WatchSource:0}: Error finding container 456c5bd2802f60577efab7f8a04a50069a09ea6dc93231573a7180fc8cc1a6dc: Status 404 returned error can't find the container with id 456c5bd2802f60577efab7f8a04a50069a09ea6dc93231573a7180fc8cc1a6dc Sep 10 05:05:09.632502 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:09.632476 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l"] Sep 10 05:05:09.638645 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:05:09.638620 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d78d427_0bc8_4271_a0b5_6c632e721ae5.slice/crio-9aed5232bbe9d205c4a0df02b1661199c85eb3dc9713c01b0f1fce996088ae63 WatchSource:0}: Error finding container 9aed5232bbe9d205c4a0df02b1661199c85eb3dc9713c01b0f1fce996088ae63: Status 404 returned error can't find the container with id 9aed5232bbe9d205c4a0df02b1661199c85eb3dc9713c01b0f1fce996088ae63 Sep 10 05:05:10.038755 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:10.038714 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" event={"ID":"8d78d427-0bc8-4271-a0b5-6c632e721ae5","Type":"ContainerStarted","Data":"115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba"} Sep 10 05:05:10.039224 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:10.038785 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" event={"ID":"8d78d427-0bc8-4271-a0b5-6c632e721ae5","Type":"ContainerStarted","Data":"9aed5232bbe9d205c4a0df02b1661199c85eb3dc9713c01b0f1fce996088ae63"} Sep 10 05:05:10.039224 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:10.038821 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:05:10.040517 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:10.040488 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" event={"ID":"1b512086-14df-402a-81e4-972c4939bfc6","Type":"ContainerStarted","Data":"2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981"} Sep 10 05:05:10.040665 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:10.040535 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" event={"ID":"1b512086-14df-402a-81e4-972c4939bfc6","Type":"ContainerStarted","Data":"456c5bd2802f60577efab7f8a04a50069a09ea6dc93231573a7180fc8cc1a6dc"} Sep 10 05:05:10.040665 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:10.040556 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:05:10.059131 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:10.059062 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" podStartSLOduration=1.059037226 podStartE2EDuration="1.059037226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:05:10.055313959 +0000 UTC m=+1429.559147020" watchObservedRunningTime="2026-09-10 05:05:10.059037226 +0000 UTC m=+1429.562870290" Sep 10 05:05:10.073073 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:10.073011 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" podStartSLOduration=2.072993516 podStartE2EDuration="2.072993516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:05:10.07020874 +0000 UTC m=+1429.574041799" watchObservedRunningTime="2026-09-10 05:05:10.072993516 +0000 UTC m=+1429.576826577" Sep 10 05:05:21.049107 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:21.049029 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:05:21.049107 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:21.049105 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:05:32.909894 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:32.909858 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s"] Sep 10 05:05:32.914107 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:32.914076 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:32.916677 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:32.916651 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-25a681-openshift-default-dockercfg-9hg9f\"" Sep 10 05:05:32.926212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:32.926180 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s"] Sep 10 05:05:33.013788 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.013742 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-certs\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.013988 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.013801 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-socket\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.013988 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.013846 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-credential-socket\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.013988 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.013870 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istiod-ca-cert\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.013988 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.013898 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-podinfo\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.014198 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.013993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-envoy\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.014198 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.014038 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-data\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.014198 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.014083 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-token\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.014198 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.014171 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6t9d\" (UniqueName: \"kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-kube-api-access-k6t9d\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.115796 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.115739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-envoy\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.116013 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.115845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-data\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.116013 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.115873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-token\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.116013 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.115915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6t9d\" (UniqueName: \"kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-kube-api-access-k6t9d\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.116013 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.115976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-certs\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.116246 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.116019 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-socket\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.116246 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.116079 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-credential-socket\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.116246 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.116106 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istiod-ca-cert\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.116246 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.116152 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-podinfo\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.116761 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.116733 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-credential-socket\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.117131 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.117100 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-data\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.117286 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.117264 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-socket\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.117423 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.117306 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istiod-ca-cert\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.117868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.117842 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-certs\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.118803 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.118778 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-envoy\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.119481 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.119442 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-podinfo\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.135537 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.135498 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-token\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.136472 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.136416 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6t9d\" (UniqueName: \"kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-kube-api-access-k6t9d\") pod \"e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.228700 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.228596 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:33.508634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.508490 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s"] Sep 10 05:05:33.511643 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:05:33.511615 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8a768b_1e52_4995_a6f1_fa8048b8ec05.slice/crio-f8463f5c3eb3d9af43183780ddacebc852ad67eea433851ed5926e7d59e5e666 WatchSource:0}: Error finding container f8463f5c3eb3d9af43183780ddacebc852ad67eea433851ed5926e7d59e5e666: Status 404 returned error can't find the container with id f8463f5c3eb3d9af43183780ddacebc852ad67eea433851ed5926e7d59e5e666 Sep 10 05:05:33.513957 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.513923 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:05:33.514034 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.513999 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:05:33.514034 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:33.514028 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:05:34.175832 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:34.175797 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" event={"ID":"7f8a768b-1e52-4995-a6f1-fa8048b8ec05","Type":"ContainerStarted","Data":"0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7"} Sep 10 05:05:34.176297 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:34.175851 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" event={"ID":"7f8a768b-1e52-4995-a6f1-fa8048b8ec05","Type":"ContainerStarted","Data":"f8463f5c3eb3d9af43183780ddacebc852ad67eea433851ed5926e7d59e5e666"} Sep 10 05:05:34.194199 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:34.194125 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" podStartSLOduration=2.194103638 podStartE2EDuration="2.194103638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:05:34.193680552 +0000 UTC m=+1453.697513624" watchObservedRunningTime="2026-09-10 05:05:34.194103638 +0000 UTC m=+1453.697936699" Sep 10 05:05:34.228805 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:34.228764 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:34.234653 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:34.234619 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:35.180846 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:35.180817 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:35.181949 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:35.181929 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:57.310074 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:57.310035 2572 generic.go:356] "Generic (PLEG): container finished" podID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerID="0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7" exitCode=0 Sep 10 05:05:57.310610 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:57.310107 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" event={"ID":"7f8a768b-1e52-4995-a6f1-fa8048b8ec05","Type":"ContainerDied","Data":"0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7"} Sep 10 05:05:57.310738 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:57.310719 2572 scope.go:117] "RemoveContainer" containerID="0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7" Sep 10 05:05:57.312110 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:57.312073 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:05:57.312187 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:57.312172 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:05:57.312241 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:57.312224 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:05:58.317839 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:58.317801 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" event={"ID":"7f8a768b-1e52-4995-a6f1-fa8048b8ec05","Type":"ContainerStarted","Data":"f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca"} Sep 10 05:05:59.229739 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:59.229691 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:05:59.229994 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:59.229961 2572 patch_prober.go:28] interesting pod/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 10 05:05:59.230055 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:05:59.230035 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 10 05:06:00.230044 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:00.229996 2572 patch_prober.go:28] interesting pod/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 10 05:06:00.230604 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:00.230080 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 10 05:06:01.229959 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:01.229921 2572 patch_prober.go:28] interesting pod/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 10 05:06:01.230164 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:01.229989 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 10 05:06:02.230052 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:02.230013 2572 patch_prober.go:28] interesting pod/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 10 05:06:02.230250 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:02.230072 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 10 05:06:03.229746 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:03.229703 2572 patch_prober.go:28] interesting pod/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 10 05:06:03.229965 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:03.229768 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 10 05:06:04.234550 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:04.234516 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:06:04.234958 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:04.234804 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:06:04.235680 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:04.235657 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:06:19.900315 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:19.900275 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l"] Sep 10 05:06:19.900965 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:19.900592 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" podUID="8d78d427-0bc8-4271-a0b5-6c632e721ae5" containerName="payload-processing" containerID="cri-o://115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba" gracePeriod=30 Sep 10 05:06:19.908357 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:19.908116 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg"] Sep 10 05:06:19.908626 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:19.908381 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" podUID="1b512086-14df-402a-81e4-972c4939bfc6" containerName="payload-pre-processing" containerID="cri-o://2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981" gracePeriod=30 Sep 10 05:06:20.205740 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.205715 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:06:20.208658 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.208634 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:06:20.230505 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.228121 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b512086-14df-402a-81e4-972c4939bfc6-plugins-config-volume\") pod \"1b512086-14df-402a-81e4-972c4939bfc6\" (UID: \"1b512086-14df-402a-81e4-972c4939bfc6\") " Sep 10 05:06:20.230505 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.228193 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d78d427-0bc8-4271-a0b5-6c632e721ae5-plugins-config-volume\") pod \"8d78d427-0bc8-4271-a0b5-6c632e721ae5\" (UID: \"8d78d427-0bc8-4271-a0b5-6c632e721ae5\") " Sep 10 05:06:20.230505 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.228238 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xgp9\" (UniqueName: \"kubernetes.io/projected/1b512086-14df-402a-81e4-972c4939bfc6-kube-api-access-6xgp9\") pod \"1b512086-14df-402a-81e4-972c4939bfc6\" (UID: \"1b512086-14df-402a-81e4-972c4939bfc6\") " Sep 10 05:06:20.230505 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.228270 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4z6n\" (UniqueName: \"kubernetes.io/projected/8d78d427-0bc8-4271-a0b5-6c632e721ae5-kube-api-access-q4z6n\") pod \"8d78d427-0bc8-4271-a0b5-6c632e721ae5\" (UID: \"8d78d427-0bc8-4271-a0b5-6c632e721ae5\") " Sep 10 05:06:20.230505 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.229442 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b512086-14df-402a-81e4-972c4939bfc6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b512086-14df-402a-81e4-972c4939bfc6" (UID: "1b512086-14df-402a-81e4-972c4939bfc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:06:20.230505 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.229934 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d78d427-0bc8-4271-a0b5-6c632e721ae5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d78d427-0bc8-4271-a0b5-6c632e721ae5" (UID: "8d78d427-0bc8-4271-a0b5-6c632e721ae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:06:20.235440 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.235140 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b512086-14df-402a-81e4-972c4939bfc6-kube-api-access-6xgp9" (OuterVolumeSpecName: "kube-api-access-6xgp9") pod "1b512086-14df-402a-81e4-972c4939bfc6" (UID: "1b512086-14df-402a-81e4-972c4939bfc6"). InnerVolumeSpecName "kube-api-access-6xgp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:06:20.235679 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.235519 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d78d427-0bc8-4271-a0b5-6c632e721ae5-kube-api-access-q4z6n" (OuterVolumeSpecName: "kube-api-access-q4z6n") pod "8d78d427-0bc8-4271-a0b5-6c632e721ae5" (UID: "8d78d427-0bc8-4271-a0b5-6c632e721ae5"). InnerVolumeSpecName "kube-api-access-q4z6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:06:20.330199 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.330160 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d78d427-0bc8-4271-a0b5-6c632e721ae5-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:20.330199 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.330197 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xgp9\" (UniqueName: \"kubernetes.io/projected/1b512086-14df-402a-81e4-972c4939bfc6-kube-api-access-6xgp9\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:20.330199 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.330207 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4z6n\" (UniqueName: \"kubernetes.io/projected/8d78d427-0bc8-4271-a0b5-6c632e721ae5-kube-api-access-q4z6n\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:20.330458 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.330216 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b512086-14df-402a-81e4-972c4939bfc6-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:20.445958 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.445869 2572 generic.go:356] "Generic (PLEG): container finished" podID="1b512086-14df-402a-81e4-972c4939bfc6" containerID="2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981" exitCode=0 Sep 10 05:06:20.445958 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.445912 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" Sep 10 05:06:20.446159 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.445954 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" event={"ID":"1b512086-14df-402a-81e4-972c4939bfc6","Type":"ContainerDied","Data":"2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981"} Sep 10 05:06:20.446159 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.446013 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg" event={"ID":"1b512086-14df-402a-81e4-972c4939bfc6","Type":"ContainerDied","Data":"456c5bd2802f60577efab7f8a04a50069a09ea6dc93231573a7180fc8cc1a6dc"} Sep 10 05:06:20.446159 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.446046 2572 scope.go:117] "RemoveContainer" containerID="2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981" Sep 10 05:06:20.447580 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.447553 2572 generic.go:356] "Generic (PLEG): container finished" podID="8d78d427-0bc8-4271-a0b5-6c632e721ae5" containerID="115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba" exitCode=0 Sep 10 05:06:20.447694 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.447583 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" event={"ID":"8d78d427-0bc8-4271-a0b5-6c632e721ae5","Type":"ContainerDied","Data":"115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba"} Sep 10 05:06:20.447694 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.447603 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" Sep 10 05:06:20.447694 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.447638 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l" event={"ID":"8d78d427-0bc8-4271-a0b5-6c632e721ae5","Type":"ContainerDied","Data":"9aed5232bbe9d205c4a0df02b1661199c85eb3dc9713c01b0f1fce996088ae63"} Sep 10 05:06:20.455251 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.455227 2572 scope.go:117] "RemoveContainer" containerID="2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981" Sep 10 05:06:20.455615 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:06:20.455595 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981\": container with ID starting with 2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981 not found: ID does not exist" containerID="2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981" Sep 10 05:06:20.455688 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.455626 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981"} err="failed to get container status \"2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981\": rpc error: code = NotFound desc = could not find container \"2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981\": container with ID starting with 2a5faba3388061b32dda44127e83709a4f80a33c7df31780a454996785b5e981 not found: ID does not exist" Sep 10 05:06:20.455688 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.455643 2572 scope.go:117] "RemoveContainer" containerID="115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba" Sep 10 05:06:20.464497 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.464478 2572 scope.go:117] "RemoveContainer" containerID="115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba" Sep 10 05:06:20.464839 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:06:20.464820 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba\": container with ID starting with 115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba not found: ID does not exist" containerID="115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba" Sep 10 05:06:20.464898 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.464847 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba"} err="failed to get container status \"115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba\": rpc error: code = NotFound desc = could not find container \"115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba\": container with ID starting with 115bfc7d540abbc299440cf55bceaf649b14f5d6bbf6b48e274253fdca7844ba not found: ID does not exist" Sep 10 05:06:20.473176 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.473147 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg"] Sep 10 05:06:20.476473 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.476431 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5e779-6f7969697d-k5vlg"] Sep 10 05:06:20.486965 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.486931 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l"] Sep 10 05:06:20.492768 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:20.492732 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5e779-6b4cd6d975-qcb7l"] Sep 10 05:06:21.036836 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:21.036801 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b512086-14df-402a-81e4-972c4939bfc6" path="/var/lib/kubelet/pods/1b512086-14df-402a-81e4-972c4939bfc6/volumes" Sep 10 05:06:21.037221 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:21.037154 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d78d427-0bc8-4271-a0b5-6c632e721ae5" path="/var/lib/kubelet/pods/8d78d427-0bc8-4271-a0b5-6c632e721ae5/volumes" Sep 10 05:06:49.591035 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:49.590935 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s"] Sep 10 05:06:49.591669 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:49.591207 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" containerID="cri-o://f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca" gracePeriod=30 Sep 10 05:06:54.852504 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.852419 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:06:54.889765 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.889727 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-socket\") pod \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " Sep 10 05:06:54.889765 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.889767 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-credential-socket\") pod \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " Sep 10 05:06:54.890015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.889798 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6t9d\" (UniqueName: \"kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-kube-api-access-k6t9d\") pod \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " Sep 10 05:06:54.890015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.889825 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-envoy\") pod \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " Sep 10 05:06:54.890015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.889849 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-certs\") pod \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " Sep 10 05:06:54.890015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.889875 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-token\") pod \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " Sep 10 05:06:54.890015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.889905 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-data\") pod \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " Sep 10 05:06:54.890015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.889951 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istiod-ca-cert\") pod \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " Sep 10 05:06:54.890333 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.890026 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-podinfo\") pod \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\" (UID: \"7f8a768b-1e52-4995-a6f1-fa8048b8ec05\") " Sep 10 05:06:54.890333 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.890047 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f8a768b-1e52-4995-a6f1-fa8048b8ec05" (UID: "7f8a768b-1e52-4995-a6f1-fa8048b8ec05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:06:54.890333 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.890114 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f8a768b-1e52-4995-a6f1-fa8048b8ec05" (UID: "7f8a768b-1e52-4995-a6f1-fa8048b8ec05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:06:54.890333 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.890213 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f8a768b-1e52-4995-a6f1-fa8048b8ec05" (UID: "7f8a768b-1e52-4995-a6f1-fa8048b8ec05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:06:54.890333 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.890225 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f8a768b-1e52-4995-a6f1-fa8048b8ec05" (UID: "7f8a768b-1e52-4995-a6f1-fa8048b8ec05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:06:54.890653 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.890468 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-data\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:54.890653 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.890481 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f8a768b-1e52-4995-a6f1-fa8048b8ec05" (UID: "7f8a768b-1e52-4995-a6f1-fa8048b8ec05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:06:54.890653 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.890496 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:54.890653 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.890514 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-credential-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:54.890653 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.890529 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-workload-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:54.892188 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.892152 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-kube-api-access-k6t9d" (OuterVolumeSpecName: "kube-api-access-k6t9d") pod "7f8a768b-1e52-4995-a6f1-fa8048b8ec05" (UID: "7f8a768b-1e52-4995-a6f1-fa8048b8ec05"). InnerVolumeSpecName "kube-api-access-k6t9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:06:54.892321 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.892235 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f8a768b-1e52-4995-a6f1-fa8048b8ec05" (UID: "7f8a768b-1e52-4995-a6f1-fa8048b8ec05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:06:54.892321 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.892263 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f8a768b-1e52-4995-a6f1-fa8048b8ec05" (UID: "7f8a768b-1e52-4995-a6f1-fa8048b8ec05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 05:06:54.892713 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.892696 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f8a768b-1e52-4995-a6f1-fa8048b8ec05" (UID: "7f8a768b-1e52-4995-a6f1-fa8048b8ec05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:06:54.991557 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.991518 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6t9d\" (UniqueName: \"kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-kube-api-access-k6t9d\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:54.991557 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.991554 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-envoy\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:54.991780 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.991569 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-token\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:54.991780 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.991580 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istiod-ca-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:54.991780 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:54.991591 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8a768b-1e52-4995-a6f1-fa8048b8ec05-istio-podinfo\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:06:55.636308 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.636272 2572 generic.go:356] "Generic (PLEG): container finished" podID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerID="f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca" exitCode=0 Sep 10 05:06:55.636308 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.636313 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" event={"ID":"7f8a768b-1e52-4995-a6f1-fa8048b8ec05","Type":"ContainerDied","Data":"f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca"} Sep 10 05:06:55.636621 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.636355 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" event={"ID":"7f8a768b-1e52-4995-a6f1-fa8048b8ec05","Type":"ContainerDied","Data":"f8463f5c3eb3d9af43183780ddacebc852ad67eea433851ed5926e7d59e5e666"} Sep 10 05:06:55.636621 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.636353 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s" Sep 10 05:06:55.636621 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.636424 2572 scope.go:117] "RemoveContainer" containerID="f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca" Sep 10 05:06:55.645660 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.645638 2572 scope.go:117] "RemoveContainer" containerID="0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7" Sep 10 05:06:55.654590 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.654563 2572 scope.go:117] "RemoveContainer" containerID="f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca" Sep 10 05:06:55.654941 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:06:55.654919 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca\": container with ID starting with f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca not found: ID does not exist" containerID="f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca" Sep 10 05:06:55.655017 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.654950 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca"} err="failed to get container status \"f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca\": rpc error: code = NotFound desc = could not find container \"f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca\": container with ID starting with f22d079d9a5eb2ef57ec1da9449b94c4b443c3db94df3eeab7c5a81939e85fca not found: ID does not exist" Sep 10 05:06:55.655017 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.654970 2572 scope.go:117] "RemoveContainer" containerID="0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7" Sep 10 05:06:55.655255 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:06:55.655237 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7\": container with ID starting with 0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7 not found: ID does not exist" containerID="0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7" Sep 10 05:06:55.655301 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.655263 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7"} err="failed to get container status \"0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7\": rpc error: code = NotFound desc = could not find container \"0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7\": container with ID starting with 0c92c658aa0ed426bcc9d0300d6bcb2cbaaa0de21fc3b34dbacf1cbdad5fb3b7 not found: ID does not exist" Sep 10 05:06:55.697260 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.697221 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s"] Sep 10 05:06:55.702069 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:55.702038 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-25a681-openshift-default-6dbf568b48-gnr8s"] Sep 10 05:06:57.037219 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:06:57.037183 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" path="/var/lib/kubelet/pods/7f8a768b-1e52-4995-a6f1-fa8048b8ec05/volumes" Sep 10 05:07:00.561389 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.561351 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn"] Sep 10 05:07:00.561928 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.561677 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" podUID="a279e799-9e46-41a2-8063-d7980773f91a" containerName="maas-api" containerID="cri-o://a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8" gracePeriod=30 Sep 10 05:07:00.623052 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.623014 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw"] Sep 10 05:07:00.623438 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.623288 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" podUID="f27cc422-1f3d-486c-b9a8-3f85495223f2" containerName="payload-processing" containerID="cri-o://c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a" gracePeriod=30 Sep 10 05:07:00.630711 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.630677 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq"] Sep 10 05:07:00.630972 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.630906 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" podUID="bcc2f1ba-0d36-4d7b-aec8-156383da5f7b" containerName="payload-pre-processing" containerID="cri-o://4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4" gracePeriod=30 Sep 10 05:07:00.903738 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.903703 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:07:00.934833 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.934800 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:07:00.949897 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.949860 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:07:00.954076 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.953673 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm27s\" (UniqueName: \"kubernetes.io/projected/f27cc422-1f3d-486c-b9a8-3f85495223f2-kube-api-access-xm27s\") pod \"f27cc422-1f3d-486c-b9a8-3f85495223f2\" (UID: \"f27cc422-1f3d-486c-b9a8-3f85495223f2\") " Sep 10 05:07:00.954076 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.953783 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27cc422-1f3d-486c-b9a8-3f85495223f2-plugins-config-volume\") pod \"f27cc422-1f3d-486c-b9a8-3f85495223f2\" (UID: \"f27cc422-1f3d-486c-b9a8-3f85495223f2\") " Sep 10 05:07:00.954861 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.954574 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27cc422-1f3d-486c-b9a8-3f85495223f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f27cc422-1f3d-486c-b9a8-3f85495223f2" (UID: "f27cc422-1f3d-486c-b9a8-3f85495223f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:07:00.956503 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:00.956475 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27cc422-1f3d-486c-b9a8-3f85495223f2-kube-api-access-xm27s" (OuterVolumeSpecName: "kube-api-access-xm27s") pod "f27cc422-1f3d-486c-b9a8-3f85495223f2" (UID: "f27cc422-1f3d-486c-b9a8-3f85495223f2"). InnerVolumeSpecName "kube-api-access-xm27s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:07:01.055533 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.055502 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a279e799-9e46-41a2-8063-d7980773f91a-openshift-service-ca\") pod \"a279e799-9e46-41a2-8063-d7980773f91a\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " Sep 10 05:07:01.055724 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.055545 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-plugins-config-volume\") pod \"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b\" (UID: \"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b\") " Sep 10 05:07:01.055724 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.055568 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbxqg\" (UniqueName: \"kubernetes.io/projected/a279e799-9e46-41a2-8063-d7980773f91a-kube-api-access-nbxqg\") pod \"a279e799-9e46-41a2-8063-d7980773f91a\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " Sep 10 05:07:01.055724 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.055587 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-maas-api-tls\") pod \"a279e799-9e46-41a2-8063-d7980773f91a\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " Sep 10 05:07:01.055724 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.055627 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-metrics-certs\") pod \"a279e799-9e46-41a2-8063-d7980773f91a\" (UID: \"a279e799-9e46-41a2-8063-d7980773f91a\") " Sep 10 05:07:01.055724 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.055701 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk4tq\" (UniqueName: \"kubernetes.io/projected/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-kube-api-access-mk4tq\") pod \"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b\" (UID: \"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b\") " Sep 10 05:07:01.055998 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.055917 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a279e799-9e46-41a2-8063-d7980773f91a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a279e799-9e46-41a2-8063-d7980773f91a" (UID: "a279e799-9e46-41a2-8063-d7980773f91a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:07:01.055998 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.055947 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bcc2f1ba-0d36-4d7b-aec8-156383da5f7b" (UID: "bcc2f1ba-0d36-4d7b-aec8-156383da5f7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:07:01.056171 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.056147 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27cc422-1f3d-486c-b9a8-3f85495223f2-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:07:01.056171 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.056173 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a279e799-9e46-41a2-8063-d7980773f91a-openshift-service-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:07:01.056360 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.056187 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:07:01.056360 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.056197 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm27s\" (UniqueName: \"kubernetes.io/projected/f27cc422-1f3d-486c-b9a8-3f85495223f2-kube-api-access-xm27s\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:07:01.057871 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.057840 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-kube-api-access-mk4tq" (OuterVolumeSpecName: "kube-api-access-mk4tq") pod "bcc2f1ba-0d36-4d7b-aec8-156383da5f7b" (UID: "bcc2f1ba-0d36-4d7b-aec8-156383da5f7b"). InnerVolumeSpecName "kube-api-access-mk4tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:07:01.057871 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.057862 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a279e799-9e46-41a2-8063-d7980773f91a" (UID: "a279e799-9e46-41a2-8063-d7980773f91a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:07:01.058171 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.058144 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a279e799-9e46-41a2-8063-d7980773f91a" (UID: "a279e799-9e46-41a2-8063-d7980773f91a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:07:01.058234 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.058152 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a279e799-9e46-41a2-8063-d7980773f91a-kube-api-access-nbxqg" (OuterVolumeSpecName: "kube-api-access-nbxqg") pod "a279e799-9e46-41a2-8063-d7980773f91a" (UID: "a279e799-9e46-41a2-8063-d7980773f91a"). InnerVolumeSpecName "kube-api-access-nbxqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:07:01.157785 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.157690 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbxqg\" (UniqueName: \"kubernetes.io/projected/a279e799-9e46-41a2-8063-d7980773f91a-kube-api-access-nbxqg\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:07:01.157785 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.157719 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-maas-api-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:07:01.157785 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.157728 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a279e799-9e46-41a2-8063-d7980773f91a-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:07:01.157785 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.157736 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk4tq\" (UniqueName: \"kubernetes.io/projected/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b-kube-api-access-mk4tq\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:07:01.671637 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.671602 2572 generic.go:356] "Generic (PLEG): container finished" podID="a279e799-9e46-41a2-8063-d7980773f91a" containerID="a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8" exitCode=0 Sep 10 05:07:01.672046 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.671655 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" Sep 10 05:07:01.672046 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.671678 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" event={"ID":"a279e799-9e46-41a2-8063-d7980773f91a","Type":"ContainerDied","Data":"a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8"} Sep 10 05:07:01.672046 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.671728 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn" event={"ID":"a279e799-9e46-41a2-8063-d7980773f91a","Type":"ContainerDied","Data":"0d279a942ceff852c708d5a0d33f4b8bfa03cb42167d11ffd56ee1b2b42318ff"} Sep 10 05:07:01.672046 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.671754 2572 scope.go:117] "RemoveContainer" containerID="a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8" Sep 10 05:07:01.673142 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.673087 2572 generic.go:356] "Generic (PLEG): container finished" podID="bcc2f1ba-0d36-4d7b-aec8-156383da5f7b" containerID="4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4" exitCode=0 Sep 10 05:07:01.673142 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.673131 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" Sep 10 05:07:01.673280 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.673173 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" event={"ID":"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b","Type":"ContainerDied","Data":"4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4"} Sep 10 05:07:01.673280 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.673226 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq" event={"ID":"bcc2f1ba-0d36-4d7b-aec8-156383da5f7b","Type":"ContainerDied","Data":"f82ca10f44d4bd9304c3b90e963bef66ede81c612038d7f134e064b8a738bd54"} Sep 10 05:07:01.674670 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.674649 2572 generic.go:356] "Generic (PLEG): container finished" podID="f27cc422-1f3d-486c-b9a8-3f85495223f2" containerID="c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a" exitCode=0 Sep 10 05:07:01.674775 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.674684 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" event={"ID":"f27cc422-1f3d-486c-b9a8-3f85495223f2","Type":"ContainerDied","Data":"c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a"} Sep 10 05:07:01.674775 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.674712 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" Sep 10 05:07:01.674775 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.674741 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw" event={"ID":"f27cc422-1f3d-486c-b9a8-3f85495223f2","Type":"ContainerDied","Data":"de35e9bf646cd72032c4644503e787e832d236aeb26b8be51a1827c067939839"} Sep 10 05:07:01.684733 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.684601 2572 scope.go:117] "RemoveContainer" containerID="a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8" Sep 10 05:07:01.685112 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:07:01.685088 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8\": container with ID starting with a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8 not found: ID does not exist" containerID="a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8" Sep 10 05:07:01.685205 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.685120 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8"} err="failed to get container status \"a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8\": rpc error: code = NotFound desc = could not find container \"a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8\": container with ID starting with a3d5f7a7d22e14c65bb5af8f169eade1b372309cb83c05962fec2424ccb3a6d8 not found: ID does not exist" Sep 10 05:07:01.685205 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.685139 2572 scope.go:117] "RemoveContainer" containerID="4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4" Sep 10 05:07:01.694098 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.694075 2572 scope.go:117] "RemoveContainer" containerID="4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4" Sep 10 05:07:01.694484 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:07:01.694438 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4\": container with ID starting with 4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4 not found: ID does not exist" containerID="4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4" Sep 10 05:07:01.694543 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.694495 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4"} err="failed to get container status \"4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4\": rpc error: code = NotFound desc = could not find container \"4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4\": container with ID starting with 4c51405b0bf3e2aac5e759c6d2a3a10fb464f91300a17039e1d0b04bb6ddb9f4 not found: ID does not exist" Sep 10 05:07:01.694543 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.694512 2572 scope.go:117] "RemoveContainer" containerID="c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a" Sep 10 05:07:01.704044 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.704004 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw"] Sep 10 05:07:01.705087 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.705062 2572 scope.go:117] "RemoveContainer" containerID="c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a" Sep 10 05:07:01.705424 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:07:01.705397 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a\": container with ID starting with c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a not found: ID does not exist" containerID="c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a" Sep 10 05:07:01.705535 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.705432 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a"} err="failed to get container status \"c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a\": rpc error: code = NotFound desc = could not find container \"c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a\": container with ID starting with c14539fb9922db510b03cbe82e6cd60356b79cb4c9633eaa9593dfc846bf481a not found: ID does not exist" Sep 10 05:07:01.706935 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.706913 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-25bd47-6fbdb78469-cktlw"] Sep 10 05:07:01.718051 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.718016 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq"] Sep 10 05:07:01.725192 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.725157 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25bd47-9b4cbf994-kshtq"] Sep 10 05:07:01.738426 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.738388 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn"] Sep 10 05:07:01.742716 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:01.742681 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-25bd47-854f8b6fdd-pnxhn"] Sep 10 05:07:03.036598 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:03.036558 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a279e799-9e46-41a2-8063-d7980773f91a" path="/var/lib/kubelet/pods/a279e799-9e46-41a2-8063-d7980773f91a/volumes" Sep 10 05:07:03.036979 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:03.036962 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcc2f1ba-0d36-4d7b-aec8-156383da5f7b" path="/var/lib/kubelet/pods/bcc2f1ba-0d36-4d7b-aec8-156383da5f7b/volumes" Sep 10 05:07:03.037283 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:03.037269 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f27cc422-1f3d-486c-b9a8-3f85495223f2" path="/var/lib/kubelet/pods/f27cc422-1f3d-486c-b9a8-3f85495223f2/volumes" Sep 10 05:07:09.994071 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994031 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp"] Sep 10 05:07:09.994751 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994696 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcc2f1ba-0d36-4d7b-aec8-156383da5f7b" containerName="payload-pre-processing" Sep 10 05:07:09.994751 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994719 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcc2f1ba-0d36-4d7b-aec8-156383da5f7b" containerName="payload-pre-processing" Sep 10 05:07:09.994751 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994740 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a279e799-9e46-41a2-8063-d7980773f91a" containerName="maas-api" Sep 10 05:07:09.994751 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994749 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a279e799-9e46-41a2-8063-d7980773f91a" containerName="maas-api" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994765 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994774 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994785 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d78d427-0bc8-4271-a0b5-6c632e721ae5" containerName="payload-processing" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994794 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d78d427-0bc8-4271-a0b5-6c632e721ae5" containerName="payload-processing" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994818 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b512086-14df-402a-81e4-972c4939bfc6" containerName="payload-pre-processing" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994827 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b512086-14df-402a-81e4-972c4939bfc6" containerName="payload-pre-processing" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994844 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27cc422-1f3d-486c-b9a8-3f85495223f2" containerName="payload-processing" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994853 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27cc422-1f3d-486c-b9a8-3f85495223f2" containerName="payload-processing" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994953 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b512086-14df-402a-81e4-972c4939bfc6" containerName="payload-pre-processing" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994968 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994979 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a279e799-9e46-41a2-8063-d7980773f91a" containerName="maas-api" Sep 10 05:07:09.994990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.994989 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" Sep 10 05:07:09.995578 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.995000 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27cc422-1f3d-486c-b9a8-3f85495223f2" containerName="payload-processing" Sep 10 05:07:09.995578 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.995009 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d78d427-0bc8-4271-a0b5-6c632e721ae5" containerName="payload-processing" Sep 10 05:07:09.995578 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:09.995019 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcc2f1ba-0d36-4d7b-aec8-156383da5f7b" containerName="payload-pre-processing" Sep 10 05:07:10.000221 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.000197 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.003196 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.003172 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"openshift-service-ca.crt\"" Sep 10 05:07:10.004538 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.004515 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"kube-root-ca.crt\"" Sep 10 05:07:10.004538 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.004535 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"rate-test-model-3b702d-kserve-self-signed-certs\"" Sep 10 05:07:10.004715 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.004517 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3b702d\"/\"default-dockercfg-2ln7j\"" Sep 10 05:07:10.009321 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.009297 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp"] Sep 10 05:07:10.147554 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.147515 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8d5s\" (UniqueName: \"kubernetes.io/projected/50cb667d-cf53-4e03-9151-87776b4271df-kube-api-access-b8d5s\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.147742 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.147577 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-model-cache\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.147742 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.147610 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-dshm\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.147742 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.147645 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-home\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.147742 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.147724 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cb667d-cf53-4e03-9151-87776b4271df-tls-certs\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.147894 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.147767 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-tmp-dir\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.248934 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.248838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8d5s\" (UniqueName: \"kubernetes.io/projected/50cb667d-cf53-4e03-9151-87776b4271df-kube-api-access-b8d5s\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.248934 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.248908 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-model-cache\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.248934 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.248934 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-dshm\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.249230 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.248956 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-home\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.249230 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.248981 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cb667d-cf53-4e03-9151-87776b4271df-tls-certs\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.249230 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.249019 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-tmp-dir\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.249436 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.249415 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-tmp-dir\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.249436 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.249431 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-home\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.249539 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.249517 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-model-cache\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.251375 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.251333 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-dshm\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.251791 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.251766 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cb667d-cf53-4e03-9151-87776b4271df-tls-certs\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.256955 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.256926 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8d5s\" (UniqueName: \"kubernetes.io/projected/50cb667d-cf53-4e03-9151-87776b4271df-kube-api-access-b8d5s\") pod \"rate-test-model-3b702d-kserve-84bbb7799-8nspp\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.313141 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.313096 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:10.587304 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.587278 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp"] Sep 10 05:07:10.589855 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:07:10.589821 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50cb667d_cf53_4e03_9151_87776b4271df.slice/crio-d3ee6d1eea4b6e3672460d4829d4f20d5c6e31e811af969792014dba07e111b2 WatchSource:0}: Error finding container d3ee6d1eea4b6e3672460d4829d4f20d5c6e31e811af969792014dba07e111b2: Status 404 returned error can't find the container with id d3ee6d1eea4b6e3672460d4829d4f20d5c6e31e811af969792014dba07e111b2 Sep 10 05:07:10.727901 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.727869 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" event={"ID":"50cb667d-cf53-4e03-9151-87776b4271df","Type":"ContainerStarted","Data":"ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562"} Sep 10 05:07:10.728076 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.727909 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" event={"ID":"50cb667d-cf53-4e03-9151-87776b4271df","Type":"ContainerStarted","Data":"d3ee6d1eea4b6e3672460d4829d4f20d5c6e31e811af969792014dba07e111b2"} Sep 10 05:07:10.746554 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:10.746497 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" podStartSLOduration=1.74648117 podStartE2EDuration="1.74648117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:07:10.745693001 +0000 UTC m=+1550.249526110" watchObservedRunningTime="2026-09-10 05:07:10.74648117 +0000 UTC m=+1550.250314231" Sep 10 05:07:18.450063 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.450027 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct"] Sep 10 05:07:18.450570 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.450304 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" podUID="cd273e3b-9056-4f71-b567-9aba987d9788" containerName="payload-pre-processing" containerID="cri-o://c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1" gracePeriod=30 Sep 10 05:07:18.704781 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.704714 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:07:18.772989 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.772948 2572 generic.go:356] "Generic (PLEG): container finished" podID="cd273e3b-9056-4f71-b567-9aba987d9788" containerID="c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1" exitCode=0 Sep 10 05:07:18.772989 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.772987 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" Sep 10 05:07:18.773211 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.773007 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" event={"ID":"cd273e3b-9056-4f71-b567-9aba987d9788","Type":"ContainerDied","Data":"c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1"} Sep 10 05:07:18.773211 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.773061 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct" event={"ID":"cd273e3b-9056-4f71-b567-9aba987d9788","Type":"ContainerDied","Data":"9eb56c4ff009c0cc65a07d7497530904bdf551304e2369983acdc5fc0f11d38f"} Sep 10 05:07:18.773211 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.773090 2572 scope.go:117] "RemoveContainer" containerID="c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1" Sep 10 05:07:18.785908 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.785883 2572 scope.go:117] "RemoveContainer" containerID="c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1" Sep 10 05:07:18.786293 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:07:18.786270 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1\": container with ID starting with c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1 not found: ID does not exist" containerID="c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1" Sep 10 05:07:18.786393 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.786301 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1"} err="failed to get container status \"c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1\": rpc error: code = NotFound desc = could not find container \"c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1\": container with ID starting with c96b2b929b6cb232f74a3ff38ed7e014b95c69359577dc67614bb8579c5274b1 not found: ID does not exist" Sep 10 05:07:18.848733 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.848694 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd273e3b-9056-4f71-b567-9aba987d9788-plugins-config-volume\") pod \"cd273e3b-9056-4f71-b567-9aba987d9788\" (UID: \"cd273e3b-9056-4f71-b567-9aba987d9788\") " Sep 10 05:07:18.848911 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.848814 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6z4f\" (UniqueName: \"kubernetes.io/projected/cd273e3b-9056-4f71-b567-9aba987d9788-kube-api-access-c6z4f\") pod \"cd273e3b-9056-4f71-b567-9aba987d9788\" (UID: \"cd273e3b-9056-4f71-b567-9aba987d9788\") " Sep 10 05:07:18.849115 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.849093 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd273e3b-9056-4f71-b567-9aba987d9788-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd273e3b-9056-4f71-b567-9aba987d9788" (UID: "cd273e3b-9056-4f71-b567-9aba987d9788"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:07:18.851081 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.851054 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd273e3b-9056-4f71-b567-9aba987d9788-kube-api-access-c6z4f" (OuterVolumeSpecName: "kube-api-access-c6z4f") pod "cd273e3b-9056-4f71-b567-9aba987d9788" (UID: "cd273e3b-9056-4f71-b567-9aba987d9788"). InnerVolumeSpecName "kube-api-access-c6z4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:07:18.950189 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.950149 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd273e3b-9056-4f71-b567-9aba987d9788-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:07:18.950189 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:18.950184 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6z4f\" (UniqueName: \"kubernetes.io/projected/cd273e3b-9056-4f71-b567-9aba987d9788-kube-api-access-c6z4f\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:07:19.090932 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:19.090896 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct"] Sep 10 05:07:19.097132 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:19.097104 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-97ffde-5bc556b986-jsrct"] Sep 10 05:07:20.313943 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:20.313898 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:20.314387 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:20.313965 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:20.326491 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:20.326435 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:20.796503 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:20.796471 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:07:21.037549 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:21.037515 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd273e3b-9056-4f71-b567-9aba987d9788" path="/var/lib/kubelet/pods/cd273e3b-9056-4f71-b567-9aba987d9788/volumes" Sep 10 05:07:54.485141 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:54.485102 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp"] Sep 10 05:07:59.399558 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.399519 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j"] Sep 10 05:07:59.399942 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.399927 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd273e3b-9056-4f71-b567-9aba987d9788" containerName="payload-pre-processing" Sep 10 05:07:59.399987 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.399944 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd273e3b-9056-4f71-b567-9aba987d9788" containerName="payload-pre-processing" Sep 10 05:07:59.399987 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.399966 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" Sep 10 05:07:59.399987 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.399975 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f8a768b-1e52-4995-a6f1-fa8048b8ec05" containerName="istio-proxy" Sep 10 05:07:59.400117 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.400104 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd273e3b-9056-4f71-b567-9aba987d9788" containerName="payload-pre-processing" Sep 10 05:07:59.403323 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.403300 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:07:59.405875 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.405852 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-95427a-serving-cert\"" Sep 10 05:07:59.415518 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.415487 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j"] Sep 10 05:07:59.548929 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.548880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f3e7af-c185-4853-a967-df7c5e752669-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:07:59.549133 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.548963 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc6h9\" (UniqueName: \"kubernetes.io/projected/46f3e7af-c185-4853-a967-df7c5e752669-kube-api-access-lc6h9\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:07:59.549133 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.549006 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:07:59.549133 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.549035 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-metrics-certs\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:07:59.650002 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.649895 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:07:59.650002 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.649948 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-metrics-certs\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:07:59.650270 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:07:59.650079 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-serving-cert: secret "maas-api-e2e-ipp-a-95427a-serving-cert" not found Sep 10 05:07:59.650270 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.650111 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f3e7af-c185-4853-a967-df7c5e752669-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:07:59.650270 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:07:59.650171 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-maas-api-tls podName:46f3e7af-c185-4853-a967-df7c5e752669 nodeName:}" failed. No retries permitted until 2026-09-10 05:08:00.150148155 +0000 UTC m=+1599.653981217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-maas-api-tls") pod "maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" (UID: "46f3e7af-c185-4853-a967-df7c5e752669") : secret "maas-api-e2e-ipp-a-95427a-serving-cert" not found Sep 10 05:07:59.650270 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.650230 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc6h9\" (UniqueName: \"kubernetes.io/projected/46f3e7af-c185-4853-a967-df7c5e752669-kube-api-access-lc6h9\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:07:59.650829 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.650809 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f3e7af-c185-4853-a967-df7c5e752669-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:07:59.652566 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.652548 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-metrics-certs\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:07:59.658970 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:07:59.658934 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc6h9\" (UniqueName: \"kubernetes.io/projected/46f3e7af-c185-4853-a967-df7c5e752669-kube-api-access-lc6h9\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:08:00.039224 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.039188 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5"] Sep 10 05:08:00.042979 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.042951 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:08:00.045675 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.045646 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-95427a\"" Sep 10 05:08:00.055567 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.055536 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5"] Sep 10 05:08:00.157084 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.157033 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b49cx\" (UniqueName: \"kubernetes.io/projected/efbd9b4e-da25-4fcf-8d69-c45d1c879767-kube-api-access-b49cx\") pod \"payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5\" (UID: \"efbd9b4e-da25-4fcf-8d69-c45d1c879767\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:08:00.157434 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.157414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbd9b4e-da25-4fcf-8d69-c45d1c879767-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5\" (UID: \"efbd9b4e-da25-4fcf-8d69-c45d1c879767\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:08:00.157731 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.157715 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:08:00.161886 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.161843 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:08:00.259531 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.259490 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b49cx\" (UniqueName: \"kubernetes.io/projected/efbd9b4e-da25-4fcf-8d69-c45d1c879767-kube-api-access-b49cx\") pod \"payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5\" (UID: \"efbd9b4e-da25-4fcf-8d69-c45d1c879767\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:08:00.259729 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.259551 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbd9b4e-da25-4fcf-8d69-c45d1c879767-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5\" (UID: \"efbd9b4e-da25-4fcf-8d69-c45d1c879767\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:08:00.260216 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.260196 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbd9b4e-da25-4fcf-8d69-c45d1c879767-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5\" (UID: \"efbd9b4e-da25-4fcf-8d69-c45d1c879767\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:08:00.268136 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.268101 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b49cx\" (UniqueName: \"kubernetes.io/projected/efbd9b4e-da25-4fcf-8d69-c45d1c879767-kube-api-access-b49cx\") pod \"payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5\" (UID: \"efbd9b4e-da25-4fcf-8d69-c45d1c879767\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:08:00.315235 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.315141 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:08:00.355670 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.355629 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:08:00.592559 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.592531 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j"] Sep 10 05:08:00.594772 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:08:00.594744 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f3e7af_c185_4853_a967_df7c5e752669.slice/crio-4e2c5be403aa29a18a818a2796a72da9f99ec1b19093f17df782c471629453c7 WatchSource:0}: Error finding container 4e2c5be403aa29a18a818a2796a72da9f99ec1b19093f17df782c471629453c7: Status 404 returned error can't find the container with id 4e2c5be403aa29a18a818a2796a72da9f99ec1b19093f17df782c471629453c7 Sep 10 05:08:00.596154 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.596131 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 05:08:00.644385 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:00.644347 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5"] Sep 10 05:08:00.647701 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:08:00.647670 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbd9b4e_da25_4fcf_8d69_c45d1c879767.slice/crio-94c17b427748703016ba27827ad9de477a9edec20a6ce209b8f7394e37be6dac WatchSource:0}: Error finding container 94c17b427748703016ba27827ad9de477a9edec20a6ce209b8f7394e37be6dac: Status 404 returned error can't find the container with id 94c17b427748703016ba27827ad9de477a9edec20a6ce209b8f7394e37be6dac Sep 10 05:08:01.006854 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:01.006797 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" event={"ID":"efbd9b4e-da25-4fcf-8d69-c45d1c879767","Type":"ContainerStarted","Data":"3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb"} Sep 10 05:08:01.007057 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:01.006863 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" event={"ID":"efbd9b4e-da25-4fcf-8d69-c45d1c879767","Type":"ContainerStarted","Data":"94c17b427748703016ba27827ad9de477a9edec20a6ce209b8f7394e37be6dac"} Sep 10 05:08:01.007057 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:01.006893 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:08:01.008081 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:01.008052 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" event={"ID":"46f3e7af-c185-4853-a967-df7c5e752669","Type":"ContainerStarted","Data":"4e2c5be403aa29a18a818a2796a72da9f99ec1b19093f17df782c471629453c7"} Sep 10 05:08:01.024976 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:01.024910 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" podStartSLOduration=1.02488802 podStartE2EDuration="1.02488802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:08:01.02414941 +0000 UTC m=+1600.527982472" watchObservedRunningTime="2026-09-10 05:08:01.02488802 +0000 UTC m=+1600.528721083" Sep 10 05:08:03.020164 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:03.020124 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" event={"ID":"46f3e7af-c185-4853-a967-df7c5e752669","Type":"ContainerStarted","Data":"20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb"} Sep 10 05:08:03.020586 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:03.020250 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:08:03.044512 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:03.044443 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" podStartSLOduration=2.359166675 podStartE2EDuration="4.044425415s" totalImagesPullingTime="1.68525874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:08:03.043151977 +0000 UTC m=+1602.546985038" watchObservedRunningTime="2026-09-10 05:08:03.044425415 +0000 UTC m=+1602.548258479" Sep 10 05:08:09.036696 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.036666 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:08:09.505680 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.505634 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" podUID="50cb667d-cf53-4e03-9151-87776b4271df" containerName="main" containerID="cri-o://ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562" gracePeriod=45 Sep 10 05:08:09.765651 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.765576 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:08:09.842229 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.842177 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-dshm\") pod \"50cb667d-cf53-4e03-9151-87776b4271df\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " Sep 10 05:08:09.842422 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.842245 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-model-cache\") pod \"50cb667d-cf53-4e03-9151-87776b4271df\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " Sep 10 05:08:09.842422 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.842273 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-home\") pod \"50cb667d-cf53-4e03-9151-87776b4271df\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " Sep 10 05:08:09.842422 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.842322 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8d5s\" (UniqueName: \"kubernetes.io/projected/50cb667d-cf53-4e03-9151-87776b4271df-kube-api-access-b8d5s\") pod \"50cb667d-cf53-4e03-9151-87776b4271df\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " Sep 10 05:08:09.842630 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.842442 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cb667d-cf53-4e03-9151-87776b4271df-tls-certs\") pod \"50cb667d-cf53-4e03-9151-87776b4271df\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " Sep 10 05:08:09.842630 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.842513 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-tmp-dir\") pod \"50cb667d-cf53-4e03-9151-87776b4271df\" (UID: \"50cb667d-cf53-4e03-9151-87776b4271df\") " Sep 10 05:08:09.842630 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.842558 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-model-cache" (OuterVolumeSpecName: "model-cache") pod "50cb667d-cf53-4e03-9151-87776b4271df" (UID: "50cb667d-cf53-4e03-9151-87776b4271df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:08:09.842630 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.842569 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-home" (OuterVolumeSpecName: "home") pod "50cb667d-cf53-4e03-9151-87776b4271df" (UID: "50cb667d-cf53-4e03-9151-87776b4271df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:08:09.842847 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.842827 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-model-cache\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:08:09.842914 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.842851 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-home\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:08:09.842914 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.842847 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50cb667d-cf53-4e03-9151-87776b4271df" (UID: "50cb667d-cf53-4e03-9151-87776b4271df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:08:09.844615 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.844586 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50cb667d-cf53-4e03-9151-87776b4271df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50cb667d-cf53-4e03-9151-87776b4271df" (UID: "50cb667d-cf53-4e03-9151-87776b4271df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:08:09.844740 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.844624 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-dshm" (OuterVolumeSpecName: "dshm") pod "50cb667d-cf53-4e03-9151-87776b4271df" (UID: "50cb667d-cf53-4e03-9151-87776b4271df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:08:09.844804 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.844733 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50cb667d-cf53-4e03-9151-87776b4271df-kube-api-access-b8d5s" (OuterVolumeSpecName: "kube-api-access-b8d5s") pod "50cb667d-cf53-4e03-9151-87776b4271df" (UID: "50cb667d-cf53-4e03-9151-87776b4271df"). InnerVolumeSpecName "kube-api-access-b8d5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:08:09.943867 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.943818 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50cb667d-cf53-4e03-9151-87776b4271df-tls-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:08:09.943867 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.943856 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-tmp-dir\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:08:09.943867 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.943867 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50cb667d-cf53-4e03-9151-87776b4271df-dshm\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:08:09.943867 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:09.943878 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8d5s\" (UniqueName: \"kubernetes.io/projected/50cb667d-cf53-4e03-9151-87776b4271df-kube-api-access-b8d5s\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:08:10.058799 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:10.058699 2572 generic.go:356] "Generic (PLEG): container finished" podID="50cb667d-cf53-4e03-9151-87776b4271df" containerID="ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562" exitCode=0 Sep 10 05:08:10.058799 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:10.058757 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" Sep 10 05:08:10.058799 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:10.058764 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" event={"ID":"50cb667d-cf53-4e03-9151-87776b4271df","Type":"ContainerDied","Data":"ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562"} Sep 10 05:08:10.059320 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:10.058809 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp" event={"ID":"50cb667d-cf53-4e03-9151-87776b4271df","Type":"ContainerDied","Data":"d3ee6d1eea4b6e3672460d4829d4f20d5c6e31e811af969792014dba07e111b2"} Sep 10 05:08:10.059320 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:10.058831 2572 scope.go:117] "RemoveContainer" containerID="ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562" Sep 10 05:08:10.068254 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:10.068228 2572 scope.go:117] "RemoveContainer" containerID="ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562" Sep 10 05:08:10.068588 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:08:10.068567 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562\": container with ID starting with ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562 not found: ID does not exist" containerID="ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562" Sep 10 05:08:10.068670 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:10.068615 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562"} err="failed to get container status \"ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562\": rpc error: code = NotFound desc = could not find container \"ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562\": container with ID starting with ee2f4638e248be5a1670d89a0b149d5f941ea43936fd1cd647ffa00669925562 not found: ID does not exist" Sep 10 05:08:10.083114 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:10.083079 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp"] Sep 10 05:08:10.086554 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:10.086523 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3b702d/rate-test-model-3b702d-kserve-84bbb7799-8nspp"] Sep 10 05:08:11.037394 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:11.037351 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50cb667d-cf53-4e03-9151-87776b4271df" path="/var/lib/kubelet/pods/50cb667d-cf53-4e03-9151-87776b4271df/volumes" Sep 10 05:08:12.015438 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:12.015406 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:08:29.246810 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.246774 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld"] Sep 10 05:08:29.247301 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.247224 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50cb667d-cf53-4e03-9151-87776b4271df" containerName="main" Sep 10 05:08:29.247301 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.247236 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="50cb667d-cf53-4e03-9151-87776b4271df" containerName="main" Sep 10 05:08:29.247393 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.247308 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="50cb667d-cf53-4e03-9151-87776b4271df" containerName="main" Sep 10 05:08:29.250622 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.250601 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:08:29.253325 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.253300 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1c7644\"" Sep 10 05:08:29.253766 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.253750 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1c7644-dockercfg-6lndh\"" Sep 10 05:08:29.261761 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.261728 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld"] Sep 10 05:08:29.266968 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.266941 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh"] Sep 10 05:08:29.270791 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.270766 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:08:29.287124 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.287091 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh"] Sep 10 05:08:29.345759 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.345711 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2fe768-6e2a-4c37-a10b-3254bef1114b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh\" (UID: \"eb2fe768-6e2a-4c37-a10b-3254bef1114b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:08:29.346006 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.345973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npr6f\" (UniqueName: \"kubernetes.io/projected/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-kube-api-access-npr6f\") pod \"payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld\" (UID: \"28e640c0-fd7b-40ec-8b0b-9432b3d536e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:08:29.346160 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.346013 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4t2\" (UniqueName: \"kubernetes.io/projected/eb2fe768-6e2a-4c37-a10b-3254bef1114b-kube-api-access-ml4t2\") pod \"payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh\" (UID: \"eb2fe768-6e2a-4c37-a10b-3254bef1114b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:08:29.346160 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.346048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld\" (UID: \"28e640c0-fd7b-40ec-8b0b-9432b3d536e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:08:29.447632 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.447591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2fe768-6e2a-4c37-a10b-3254bef1114b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh\" (UID: \"eb2fe768-6e2a-4c37-a10b-3254bef1114b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:08:29.447825 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.447687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npr6f\" (UniqueName: \"kubernetes.io/projected/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-kube-api-access-npr6f\") pod \"payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld\" (UID: \"28e640c0-fd7b-40ec-8b0b-9432b3d536e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:08:29.447825 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.447706 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4t2\" (UniqueName: \"kubernetes.io/projected/eb2fe768-6e2a-4c37-a10b-3254bef1114b-kube-api-access-ml4t2\") pod \"payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh\" (UID: \"eb2fe768-6e2a-4c37-a10b-3254bef1114b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:08:29.447825 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.447726 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld\" (UID: \"28e640c0-fd7b-40ec-8b0b-9432b3d536e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:08:29.448411 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.448388 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2fe768-6e2a-4c37-a10b-3254bef1114b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh\" (UID: \"eb2fe768-6e2a-4c37-a10b-3254bef1114b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:08:29.448493 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.448392 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld\" (UID: \"28e640c0-fd7b-40ec-8b0b-9432b3d536e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:08:29.456213 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.456174 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4t2\" (UniqueName: \"kubernetes.io/projected/eb2fe768-6e2a-4c37-a10b-3254bef1114b-kube-api-access-ml4t2\") pod \"payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh\" (UID: \"eb2fe768-6e2a-4c37-a10b-3254bef1114b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:08:29.456370 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.456345 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npr6f\" (UniqueName: \"kubernetes.io/projected/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-kube-api-access-npr6f\") pod \"payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld\" (UID: \"28e640c0-fd7b-40ec-8b0b-9432b3d536e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:08:29.563360 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.563255 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:08:29.583973 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.583935 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:08:29.843128 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.843099 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld"] Sep 10 05:08:29.844854 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:08:29.844818 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e640c0_fd7b_40ec_8b0b_9432b3d536e7.slice/crio-d2c5f09f3187b858622a350ee8057d2c3473ad06fc01d326e04b29ee1b857016 WatchSource:0}: Error finding container d2c5f09f3187b858622a350ee8057d2c3473ad06fc01d326e04b29ee1b857016: Status 404 returned error can't find the container with id d2c5f09f3187b858622a350ee8057d2c3473ad06fc01d326e04b29ee1b857016 Sep 10 05:08:29.871807 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:29.871780 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh"] Sep 10 05:08:29.877698 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:08:29.877664 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2fe768_6e2a_4c37_a10b_3254bef1114b.slice/crio-244f0963172872f2f6152db98418a07898ccac4909ff164b07c553694e3cbb9f WatchSource:0}: Error finding container 244f0963172872f2f6152db98418a07898ccac4909ff164b07c553694e3cbb9f: Status 404 returned error can't find the container with id 244f0963172872f2f6152db98418a07898ccac4909ff164b07c553694e3cbb9f Sep 10 05:08:30.174279 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:30.174169 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" event={"ID":"eb2fe768-6e2a-4c37-a10b-3254bef1114b","Type":"ContainerStarted","Data":"782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2"} Sep 10 05:08:30.174279 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:30.174234 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" event={"ID":"eb2fe768-6e2a-4c37-a10b-3254bef1114b","Type":"ContainerStarted","Data":"244f0963172872f2f6152db98418a07898ccac4909ff164b07c553694e3cbb9f"} Sep 10 05:08:30.174279 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:30.174264 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:08:30.176179 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:30.176148 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" event={"ID":"28e640c0-fd7b-40ec-8b0b-9432b3d536e7","Type":"ContainerStarted","Data":"f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691"} Sep 10 05:08:30.176309 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:30.176198 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" event={"ID":"28e640c0-fd7b-40ec-8b0b-9432b3d536e7","Type":"ContainerStarted","Data":"d2c5f09f3187b858622a350ee8057d2c3473ad06fc01d326e04b29ee1b857016"} Sep 10 05:08:30.176309 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:30.176228 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:08:30.192922 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:30.192852 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" podStartSLOduration=1.192829154 podStartE2EDuration="1.192829154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:08:30.190873494 +0000 UTC m=+1629.694706557" watchObservedRunningTime="2026-09-10 05:08:30.192829154 +0000 UTC m=+1629.696662216" Sep 10 05:08:30.208208 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:30.208141 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" podStartSLOduration=1.20811597 podStartE2EDuration="1.20811597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:08:30.207366597 +0000 UTC m=+1629.711199666" watchObservedRunningTime="2026-09-10 05:08:30.20811597 +0000 UTC m=+1629.711949032" Sep 10 05:08:41.185131 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:41.185097 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:08:41.185560 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:41.185310 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:08:57.684023 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:57.683986 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4nxnk_cf246a19-caf3-4fd6-8611-add57e445231/payload-processing/0.log" Sep 10 05:08:57.812623 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:08:57.812592 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh_eb2fe768-6e2a-4c37-a10b-3254bef1114b/payload-processing/0.log" Sep 10 05:09:39.129891 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.129857 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4nxnk_cf246a19-caf3-4fd6-8611-add57e445231/payload-processing/0.log" Sep 10 05:09:39.268407 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.268358 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh_eb2fe768-6e2a-4c37-a10b-3254bef1114b/payload-processing/0.log" Sep 10 05:09:39.864742 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.864703 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb"] Sep 10 05:09:39.870774 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.870737 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:39.873359 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.873335 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-56fdcd-openshift-default-dockercfg-mr7qs\"" Sep 10 05:09:39.880728 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.880701 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb"] Sep 10 05:09:39.966187 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.966148 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-credential-socket\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:39.966381 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.966240 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkfr7\" (UniqueName: \"kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-kube-api-access-nkfr7\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:39.966381 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.966279 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-certs\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:39.966381 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.966313 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-podinfo\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:39.966381 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.966342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-socket\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:39.966579 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.966391 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-token\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:39.966579 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.966423 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-data\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:39.966579 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.966491 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istiod-ca-cert\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:39.966579 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:39.966552 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-envoy\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.067296 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-credential-socket\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.067540 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067344 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkfr7\" (UniqueName: \"kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-kube-api-access-nkfr7\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.067540 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-certs\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.067540 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067404 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-podinfo\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.067540 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067433 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-socket\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.067540 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067509 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-token\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.067540 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067534 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-data\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.067888 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067586 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istiod-ca-cert\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.067888 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-envoy\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.067888 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067736 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-credential-socket\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.067888 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067875 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-data\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.068059 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067932 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-certs\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.068059 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.067984 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-socket\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.068440 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.068414 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istiod-ca-cert\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.069824 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.069797 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-envoy\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.069915 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.069889 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-podinfo\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.076639 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.076614 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-token\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.076826 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.076801 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkfr7\" (UniqueName: \"kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-kube-api-access-nkfr7\") pod \"e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.183402 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.183311 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:40.464507 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.464475 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb"] Sep 10 05:09:40.470805 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.470770 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:09:40.470921 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.470844 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:09:40.470921 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.470877 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:09:40.559561 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.559516 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" event={"ID":"8570dd9c-7dec-4a4c-8a17-d5e33393c928","Type":"ContainerStarted","Data":"4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86"} Sep 10 05:09:40.559724 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.559591 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" event={"ID":"8570dd9c-7dec-4a4c-8a17-d5e33393c928","Type":"ContainerStarted","Data":"9b19c49dbb7bac5740db00a5e3fa80f029cce867d1aa402efb96889e12c9e5c1"} Sep 10 05:09:40.584409 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:40.584340 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" podStartSLOduration=1.584314056 podStartE2EDuration="1.584314056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:09:40.581263272 +0000 UTC m=+1700.085096362" watchObservedRunningTime="2026-09-10 05:09:40.584314056 +0000 UTC m=+1700.088147324" Sep 10 05:09:41.184143 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:41.184113 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:41.188757 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:41.188728 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:41.564434 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:41.564406 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:41.565328 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:41.565306 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:09:44.372477 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.372382 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr"] Sep 10 05:09:44.376299 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.376279 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:09:44.378785 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.378762 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-56fdcd\"" Sep 10 05:09:44.378911 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.378816 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-56fdcd-dockercfg-w7rkz\"" Sep 10 05:09:44.389163 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.389137 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr"] Sep 10 05:09:44.514841 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.514802 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjgrb\" (UniqueName: \"kubernetes.io/projected/cec36820-a2a3-4cdd-8c09-0c3ce792361e-kube-api-access-fjgrb\") pod \"payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr\" (UID: \"cec36820-a2a3-4cdd-8c09-0c3ce792361e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:09:44.515090 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.515028 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec36820-a2a3-4cdd-8c09-0c3ce792361e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr\" (UID: \"cec36820-a2a3-4cdd-8c09-0c3ce792361e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:09:44.615937 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.615891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec36820-a2a3-4cdd-8c09-0c3ce792361e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr\" (UID: \"cec36820-a2a3-4cdd-8c09-0c3ce792361e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:09:44.616131 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.615951 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgrb\" (UniqueName: \"kubernetes.io/projected/cec36820-a2a3-4cdd-8c09-0c3ce792361e-kube-api-access-fjgrb\") pod \"payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr\" (UID: \"cec36820-a2a3-4cdd-8c09-0c3ce792361e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:09:44.616612 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.616594 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec36820-a2a3-4cdd-8c09-0c3ce792361e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr\" (UID: \"cec36820-a2a3-4cdd-8c09-0c3ce792361e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:09:44.624546 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.624483 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjgrb\" (UniqueName: \"kubernetes.io/projected/cec36820-a2a3-4cdd-8c09-0c3ce792361e-kube-api-access-fjgrb\") pod \"payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr\" (UID: \"cec36820-a2a3-4cdd-8c09-0c3ce792361e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:09:44.688298 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.688257 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:09:44.963665 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:44.963640 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr"] Sep 10 05:09:44.966084 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:09:44.966052 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec36820_a2a3_4cdd_8c09_0c3ce792361e.slice/crio-c296bf6da33a1964f9e0f84cadca06645201d6fd3cccba943752741180e5f9bf WatchSource:0}: Error finding container c296bf6da33a1964f9e0f84cadca06645201d6fd3cccba943752741180e5f9bf: Status 404 returned error can't find the container with id c296bf6da33a1964f9e0f84cadca06645201d6fd3cccba943752741180e5f9bf Sep 10 05:09:45.587132 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:45.587088 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" event={"ID":"cec36820-a2a3-4cdd-8c09-0c3ce792361e","Type":"ContainerStarted","Data":"88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12"} Sep 10 05:09:45.587542 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:45.587148 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" event={"ID":"cec36820-a2a3-4cdd-8c09-0c3ce792361e","Type":"ContainerStarted","Data":"c296bf6da33a1964f9e0f84cadca06645201d6fd3cccba943752741180e5f9bf"} Sep 10 05:09:45.587542 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:45.587169 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:09:45.605384 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:45.605319 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" podStartSLOduration=1.605294505 podStartE2EDuration="1.605294505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:09:45.60426029 +0000 UTC m=+1705.108093352" watchObservedRunningTime="2026-09-10 05:09:45.605294505 +0000 UTC m=+1705.109127567" Sep 10 05:09:56.595999 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:09:56.595967 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:10:02.680262 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:02.680156 2572 generic.go:356] "Generic (PLEG): container finished" podID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerID="4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86" exitCode=0 Sep 10 05:10:02.680262 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:02.680234 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" event={"ID":"8570dd9c-7dec-4a4c-8a17-d5e33393c928","Type":"ContainerDied","Data":"4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86"} Sep 10 05:10:02.680746 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:02.680694 2572 scope.go:117] "RemoveContainer" containerID="4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86" Sep 10 05:10:02.681847 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:02.681823 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:10:02.681911 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:02.681889 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:10:02.681949 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:02.681919 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 05:10:03.691931 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:03.691886 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" event={"ID":"8570dd9c-7dec-4a4c-8a17-d5e33393c928","Type":"ContainerStarted","Data":"d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c"} Sep 10 05:10:04.183733 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:04.183694 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:10:04.184100 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:04.184078 2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 05:10:04.184275 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:04.184248 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 05:10:05.184016 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:05.183967 2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 05:10:05.184558 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:05.184042 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 05:10:06.184235 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:06.184183 2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 05:10:06.184782 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:06.184262 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 05:10:07.184846 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:07.184800 2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 05:10:07.185355 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:07.184878 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 05:10:08.184840 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:08.184787 2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 05:10:08.185075 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:08.184881 2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 05:10:09.188699 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:09.188663 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:10:09.189093 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:09.188932 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:10:09.189590 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:09.189574 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:10:14.424680 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.424639 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr"] Sep 10 05:10:14.425201 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.424912 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" podUID="cec36820-a2a3-4cdd-8c09-0c3ce792361e" containerName="payload-pre-processing" containerID="cri-o://88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12" gracePeriod=30 Sep 10 05:10:14.696156 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.696120 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:10:14.758945 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.758901 2572 generic.go:356] "Generic (PLEG): container finished" podID="cec36820-a2a3-4cdd-8c09-0c3ce792361e" containerID="88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12" exitCode=0 Sep 10 05:10:14.759161 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.759070 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" event={"ID":"cec36820-a2a3-4cdd-8c09-0c3ce792361e","Type":"ContainerDied","Data":"88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12"} Sep 10 05:10:14.759238 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.759172 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" event={"ID":"cec36820-a2a3-4cdd-8c09-0c3ce792361e","Type":"ContainerDied","Data":"c296bf6da33a1964f9e0f84cadca06645201d6fd3cccba943752741180e5f9bf"} Sep 10 05:10:14.759238 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.759208 2572 scope.go:117] "RemoveContainer" containerID="88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12" Sep 10 05:10:14.759390 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.759367 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr" Sep 10 05:10:14.786429 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.786395 2572 scope.go:117] "RemoveContainer" containerID="88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12" Sep 10 05:10:14.787170 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:10:14.787119 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12\": container with ID starting with 88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12 not found: ID does not exist" containerID="88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12" Sep 10 05:10:14.787296 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.787172 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12"} err="failed to get container status \"88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12\": rpc error: code = NotFound desc = could not find container \"88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12\": container with ID starting with 88c8ddce560c7be1d345f0e7f43fbc3599da1e0b88c713ea6ce296e8af08ee12 not found: ID does not exist" Sep 10 05:10:14.824873 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.824836 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec36820-a2a3-4cdd-8c09-0c3ce792361e-plugins-config-volume\") pod \"cec36820-a2a3-4cdd-8c09-0c3ce792361e\" (UID: \"cec36820-a2a3-4cdd-8c09-0c3ce792361e\") " Sep 10 05:10:14.825077 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.824884 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjgrb\" (UniqueName: \"kubernetes.io/projected/cec36820-a2a3-4cdd-8c09-0c3ce792361e-kube-api-access-fjgrb\") pod \"cec36820-a2a3-4cdd-8c09-0c3ce792361e\" (UID: \"cec36820-a2a3-4cdd-8c09-0c3ce792361e\") " Sep 10 05:10:14.825250 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.825223 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec36820-a2a3-4cdd-8c09-0c3ce792361e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cec36820-a2a3-4cdd-8c09-0c3ce792361e" (UID: "cec36820-a2a3-4cdd-8c09-0c3ce792361e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:10:14.827080 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.827052 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec36820-a2a3-4cdd-8c09-0c3ce792361e-kube-api-access-fjgrb" (OuterVolumeSpecName: "kube-api-access-fjgrb") pod "cec36820-a2a3-4cdd-8c09-0c3ce792361e" (UID: "cec36820-a2a3-4cdd-8c09-0c3ce792361e"). InnerVolumeSpecName "kube-api-access-fjgrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:10:14.925879 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.925788 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec36820-a2a3-4cdd-8c09-0c3ce792361e-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:14.925879 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:14.925820 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjgrb\" (UniqueName: \"kubernetes.io/projected/cec36820-a2a3-4cdd-8c09-0c3ce792361e-kube-api-access-fjgrb\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:15.079127 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:15.079084 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr"] Sep 10 05:10:15.082814 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:15.082784 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-56fdcd-5f765c4c4-kmzzr"] Sep 10 05:10:17.037303 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:17.037269 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cec36820-a2a3-4cdd-8c09-0c3ce792361e" path="/var/lib/kubelet/pods/cec36820-a2a3-4cdd-8c09-0c3ce792361e/volumes" Sep 10 05:10:21.101902 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:21.101860 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb"] Sep 10 05:10:21.102318 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:21.102146 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" containerID="cri-o://d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c" gracePeriod=30 Sep 10 05:10:24.191619 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:24.191576 2572 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 10 05:10:24.192011 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:24.191671 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 05:10:26.358532 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.358501 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:10:26.444050 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.443973 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-data\") pod \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " Sep 10 05:10:26.444050 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444035 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-credential-socket\") pod \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " Sep 10 05:10:26.444261 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444070 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkfr7\" (UniqueName: \"kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-kube-api-access-nkfr7\") pod \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " Sep 10 05:10:26.444261 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444111 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-podinfo\") pod \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " Sep 10 05:10:26.444261 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444171 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-certs\") pod \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " Sep 10 05:10:26.444261 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444217 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-envoy\") pod \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " Sep 10 05:10:26.444261 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444254 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-socket\") pod \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " Sep 10 05:10:26.444641 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444284 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istiod-ca-cert\") pod \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " Sep 10 05:10:26.444641 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444312 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-token\") pod \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\" (UID: \"8570dd9c-7dec-4a4c-8a17-d5e33393c928\") " Sep 10 05:10:26.444641 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444289 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-data" (OuterVolumeSpecName: "istio-data") pod "8570dd9c-7dec-4a4c-8a17-d5e33393c928" (UID: "8570dd9c-7dec-4a4c-8a17-d5e33393c928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:10:26.444641 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444351 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8570dd9c-7dec-4a4c-8a17-d5e33393c928" (UID: "8570dd9c-7dec-4a4c-8a17-d5e33393c928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:10:26.444641 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444536 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8570dd9c-7dec-4a4c-8a17-d5e33393c928" (UID: "8570dd9c-7dec-4a4c-8a17-d5e33393c928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:10:26.444641 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444567 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8570dd9c-7dec-4a4c-8a17-d5e33393c928" (UID: "8570dd9c-7dec-4a4c-8a17-d5e33393c928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:10:26.444928 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444696 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-credential-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:26.444928 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444721 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:26.444928 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444742 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-workload-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:26.444928 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444756 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-data\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:26.444928 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.444720 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8570dd9c-7dec-4a4c-8a17-d5e33393c928" (UID: "8570dd9c-7dec-4a4c-8a17-d5e33393c928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:10:26.446480 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.446441 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-token" (OuterVolumeSpecName: "istio-token") pod "8570dd9c-7dec-4a4c-8a17-d5e33393c928" (UID: "8570dd9c-7dec-4a4c-8a17-d5e33393c928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:10:26.446594 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.446515 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8570dd9c-7dec-4a4c-8a17-d5e33393c928" (UID: "8570dd9c-7dec-4a4c-8a17-d5e33393c928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:10:26.446752 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.446734 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-kube-api-access-nkfr7" (OuterVolumeSpecName: "kube-api-access-nkfr7") pod "8570dd9c-7dec-4a4c-8a17-d5e33393c928" (UID: "8570dd9c-7dec-4a4c-8a17-d5e33393c928"). InnerVolumeSpecName "kube-api-access-nkfr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:10:26.446816 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.446753 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8570dd9c-7dec-4a4c-8a17-d5e33393c928" (UID: "8570dd9c-7dec-4a4c-8a17-d5e33393c928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 05:10:26.546030 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.545984 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-envoy\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:26.546030 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.546021 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istiod-ca-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:26.546030 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.546030 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-token\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:26.546030 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.546038 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkfr7\" (UniqueName: \"kubernetes.io/projected/8570dd9c-7dec-4a4c-8a17-d5e33393c928-kube-api-access-nkfr7\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:26.546327 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.546052 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8570dd9c-7dec-4a4c-8a17-d5e33393c928-istio-podinfo\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:26.832149 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.832111 2572 generic.go:356] "Generic (PLEG): container finished" podID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerID="d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c" exitCode=0 Sep 10 05:10:26.832323 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.832156 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" Sep 10 05:10:26.832323 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.832152 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" event={"ID":"8570dd9c-7dec-4a4c-8a17-d5e33393c928","Type":"ContainerDied","Data":"d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c"} Sep 10 05:10:26.832323 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.832240 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb" event={"ID":"8570dd9c-7dec-4a4c-8a17-d5e33393c928","Type":"ContainerDied","Data":"9b19c49dbb7bac5740db00a5e3fa80f029cce867d1aa402efb96889e12c9e5c1"} Sep 10 05:10:26.832323 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.832270 2572 scope.go:117] "RemoveContainer" containerID="d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c" Sep 10 05:10:26.842420 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.842399 2572 scope.go:117] "RemoveContainer" containerID="4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86" Sep 10 05:10:26.852351 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.852328 2572 scope.go:117] "RemoveContainer" containerID="d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c" Sep 10 05:10:26.852716 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:10:26.852694 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c\": container with ID starting with d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c not found: ID does not exist" containerID="d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c" Sep 10 05:10:26.852790 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.852726 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c"} err="failed to get container status \"d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c\": rpc error: code = NotFound desc = could not find container \"d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c\": container with ID starting with d7957ff826007cd6fb60db6333135aa6afd5b4810a0d0f029df8e7df7e074f0c not found: ID does not exist" Sep 10 05:10:26.852790 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.852744 2572 scope.go:117] "RemoveContainer" containerID="4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86" Sep 10 05:10:26.853024 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:10:26.853009 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86\": container with ID starting with 4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86 not found: ID does not exist" containerID="4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86" Sep 10 05:10:26.853064 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.853028 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86"} err="failed to get container status \"4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86\": rpc error: code = NotFound desc = could not find container \"4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86\": container with ID starting with 4895e186eee09937df6fbc81b4bc9c8d5bc2c2a653257a2fb65b3909cf2d5a86 not found: ID does not exist" Sep 10 05:10:26.856491 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.856443 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb"] Sep 10 05:10:26.860629 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:26.860598 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-56fdcd-openshift-default-56bf7cfd7b-dxbbb"] Sep 10 05:10:27.037875 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:27.037841 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" path="/var/lib/kubelet/pods/8570dd9c-7dec-4a4c-8a17-d5e33393c928/volumes" Sep 10 05:10:40.444340 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.444300 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j"] Sep 10 05:10:40.444958 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.444533 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" podUID="46f3e7af-c185-4853-a967-df7c5e752669" containerName="maas-api" containerID="cri-o://20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb" gracePeriod=30 Sep 10 05:10:40.505587 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.505550 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5"] Sep 10 05:10:40.505823 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.505774 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" podUID="efbd9b4e-da25-4fcf-8d69-c45d1c879767" containerName="payload-pre-processing" containerID="cri-o://3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb" gracePeriod=30 Sep 10 05:10:40.755130 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.755096 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:10:40.791582 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.791551 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc6h9\" (UniqueName: \"kubernetes.io/projected/46f3e7af-c185-4853-a967-df7c5e752669-kube-api-access-lc6h9\") pod \"46f3e7af-c185-4853-a967-df7c5e752669\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " Sep 10 05:10:40.791782 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.791643 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-metrics-certs\") pod \"46f3e7af-c185-4853-a967-df7c5e752669\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " Sep 10 05:10:40.791782 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.791743 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-maas-api-tls\") pod \"46f3e7af-c185-4853-a967-df7c5e752669\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " Sep 10 05:10:40.791782 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.791776 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f3e7af-c185-4853-a967-df7c5e752669-openshift-service-ca\") pod \"46f3e7af-c185-4853-a967-df7c5e752669\" (UID: \"46f3e7af-c185-4853-a967-df7c5e752669\") " Sep 10 05:10:40.792519 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.792487 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f3e7af-c185-4853-a967-df7c5e752669-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46f3e7af-c185-4853-a967-df7c5e752669" (UID: "46f3e7af-c185-4853-a967-df7c5e752669"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:10:40.793890 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.793863 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "46f3e7af-c185-4853-a967-df7c5e752669" (UID: "46f3e7af-c185-4853-a967-df7c5e752669"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:10:40.794251 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.794226 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46f3e7af-c185-4853-a967-df7c5e752669" (UID: "46f3e7af-c185-4853-a967-df7c5e752669"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:10:40.794344 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.794263 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f3e7af-c185-4853-a967-df7c5e752669-kube-api-access-lc6h9" (OuterVolumeSpecName: "kube-api-access-lc6h9") pod "46f3e7af-c185-4853-a967-df7c5e752669" (UID: "46f3e7af-c185-4853-a967-df7c5e752669"). InnerVolumeSpecName "kube-api-access-lc6h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:10:40.810693 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.810645 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:10:40.892958 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.892921 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b49cx\" (UniqueName: \"kubernetes.io/projected/efbd9b4e-da25-4fcf-8d69-c45d1c879767-kube-api-access-b49cx\") pod \"efbd9b4e-da25-4fcf-8d69-c45d1c879767\" (UID: \"efbd9b4e-da25-4fcf-8d69-c45d1c879767\") " Sep 10 05:10:40.893146 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.893085 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbd9b4e-da25-4fcf-8d69-c45d1c879767-plugins-config-volume\") pod \"efbd9b4e-da25-4fcf-8d69-c45d1c879767\" (UID: \"efbd9b4e-da25-4fcf-8d69-c45d1c879767\") " Sep 10 05:10:40.893293 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.893271 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-maas-api-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:40.893293 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.893287 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46f3e7af-c185-4853-a967-df7c5e752669-openshift-service-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:40.893362 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.893296 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc6h9\" (UniqueName: \"kubernetes.io/projected/46f3e7af-c185-4853-a967-df7c5e752669-kube-api-access-lc6h9\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:40.893362 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.893304 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46f3e7af-c185-4853-a967-df7c5e752669-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:40.893481 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.893430 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbd9b4e-da25-4fcf-8d69-c45d1c879767-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "efbd9b4e-da25-4fcf-8d69-c45d1c879767" (UID: "efbd9b4e-da25-4fcf-8d69-c45d1c879767"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:10:40.895089 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.895065 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbd9b4e-da25-4fcf-8d69-c45d1c879767-kube-api-access-b49cx" (OuterVolumeSpecName: "kube-api-access-b49cx") pod "efbd9b4e-da25-4fcf-8d69-c45d1c879767" (UID: "efbd9b4e-da25-4fcf-8d69-c45d1c879767"). InnerVolumeSpecName "kube-api-access-b49cx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:10:40.922628 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.922587 2572 generic.go:356] "Generic (PLEG): container finished" podID="46f3e7af-c185-4853-a967-df7c5e752669" containerID="20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb" exitCode=0 Sep 10 05:10:40.922825 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.922648 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" Sep 10 05:10:40.922825 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.922674 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" event={"ID":"46f3e7af-c185-4853-a967-df7c5e752669","Type":"ContainerDied","Data":"20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb"} Sep 10 05:10:40.922825 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.922766 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j" event={"ID":"46f3e7af-c185-4853-a967-df7c5e752669","Type":"ContainerDied","Data":"4e2c5be403aa29a18a818a2796a72da9f99ec1b19093f17df782c471629453c7"} Sep 10 05:10:40.922825 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.922796 2572 scope.go:117] "RemoveContainer" containerID="20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb" Sep 10 05:10:40.924210 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.924182 2572 generic.go:356] "Generic (PLEG): container finished" podID="efbd9b4e-da25-4fcf-8d69-c45d1c879767" containerID="3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb" exitCode=0 Sep 10 05:10:40.924334 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.924259 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" Sep 10 05:10:40.924334 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.924213 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" event={"ID":"efbd9b4e-da25-4fcf-8d69-c45d1c879767","Type":"ContainerDied","Data":"3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb"} Sep 10 05:10:40.924437 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.924337 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5" event={"ID":"efbd9b4e-da25-4fcf-8d69-c45d1c879767","Type":"ContainerDied","Data":"94c17b427748703016ba27827ad9de477a9edec20a6ce209b8f7394e37be6dac"} Sep 10 05:10:40.932496 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.932476 2572 scope.go:117] "RemoveContainer" containerID="20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb" Sep 10 05:10:40.932862 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:10:40.932842 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb\": container with ID starting with 20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb not found: ID does not exist" containerID="20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb" Sep 10 05:10:40.932927 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.932873 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb"} err="failed to get container status \"20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb\": rpc error: code = NotFound desc = could not find container \"20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb\": container with ID starting with 20fd16b617ca085076c100bc9ee7b5ec2d6553e8f2e9fe026dbfc7faffafa6cb not found: ID does not exist" Sep 10 05:10:40.932927 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.932892 2572 scope.go:117] "RemoveContainer" containerID="3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb" Sep 10 05:10:40.941990 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.941970 2572 scope.go:117] "RemoveContainer" containerID="3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb" Sep 10 05:10:40.942430 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:10:40.942409 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb\": container with ID starting with 3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb not found: ID does not exist" containerID="3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb" Sep 10 05:10:40.942625 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.942437 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb"} err="failed to get container status \"3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb\": rpc error: code = NotFound desc = could not find container \"3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb\": container with ID starting with 3792c15062df9420f554ec5bc9284e3647bac823070547cef6b45a72fe21befb not found: ID does not exist" Sep 10 05:10:40.949943 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.949912 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j"] Sep 10 05:10:40.953943 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.953915 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95427a-67b7dc6b88-h986j"] Sep 10 05:10:40.966818 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.966779 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5"] Sep 10 05:10:40.971973 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.971882 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95427a-5ccbcb995d-gvxs5"] Sep 10 05:10:40.994059 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.994024 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efbd9b4e-da25-4fcf-8d69-c45d1c879767-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:40.994059 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:40.994057 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b49cx\" (UniqueName: \"kubernetes.io/projected/efbd9b4e-da25-4fcf-8d69-c45d1c879767-kube-api-access-b49cx\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:10:41.038349 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:41.038308 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46f3e7af-c185-4853-a967-df7c5e752669" path="/var/lib/kubelet/pods/46f3e7af-c185-4853-a967-df7c5e752669/volumes" Sep 10 05:10:41.038824 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:10:41.038802 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efbd9b4e-da25-4fcf-8d69-c45d1c879767" path="/var/lib/kubelet/pods/efbd9b4e-da25-4fcf-8d69-c45d1c879767/volumes" Sep 10 05:11:39.679512 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:39.679469 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh"] Sep 10 05:11:39.680129 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:39.679786 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" podUID="eb2fe768-6e2a-4c37-a10b-3254bef1114b" containerName="payload-processing" containerID="cri-o://782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2" gracePeriod=30 Sep 10 05:11:39.690368 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:39.690315 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld"] Sep 10 05:11:39.690637 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:39.690586 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" podUID="28e640c0-fd7b-40ec-8b0b-9432b3d536e7" containerName="payload-pre-processing" containerID="cri-o://f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691" gracePeriod=30 Sep 10 05:11:39.985938 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:39.985910 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:11:39.988950 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:39.988923 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:11:40.092609 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.092567 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-plugins-config-volume\") pod \"28e640c0-fd7b-40ec-8b0b-9432b3d536e7\" (UID: \"28e640c0-fd7b-40ec-8b0b-9432b3d536e7\") " Sep 10 05:11:40.092609 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.092611 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2fe768-6e2a-4c37-a10b-3254bef1114b-plugins-config-volume\") pod \"eb2fe768-6e2a-4c37-a10b-3254bef1114b\" (UID: \"eb2fe768-6e2a-4c37-a10b-3254bef1114b\") " Sep 10 05:11:40.092888 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.092652 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml4t2\" (UniqueName: \"kubernetes.io/projected/eb2fe768-6e2a-4c37-a10b-3254bef1114b-kube-api-access-ml4t2\") pod \"eb2fe768-6e2a-4c37-a10b-3254bef1114b\" (UID: \"eb2fe768-6e2a-4c37-a10b-3254bef1114b\") " Sep 10 05:11:40.092888 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.092702 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npr6f\" (UniqueName: \"kubernetes.io/projected/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-kube-api-access-npr6f\") pod \"28e640c0-fd7b-40ec-8b0b-9432b3d536e7\" (UID: \"28e640c0-fd7b-40ec-8b0b-9432b3d536e7\") " Sep 10 05:11:40.093060 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.093019 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28e640c0-fd7b-40ec-8b0b-9432b3d536e7" (UID: "28e640c0-fd7b-40ec-8b0b-9432b3d536e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:11:40.093176 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.093072 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2fe768-6e2a-4c37-a10b-3254bef1114b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb2fe768-6e2a-4c37-a10b-3254bef1114b" (UID: "eb2fe768-6e2a-4c37-a10b-3254bef1114b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:11:40.093176 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.093133 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:11:40.094923 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.094896 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2fe768-6e2a-4c37-a10b-3254bef1114b-kube-api-access-ml4t2" (OuterVolumeSpecName: "kube-api-access-ml4t2") pod "eb2fe768-6e2a-4c37-a10b-3254bef1114b" (UID: "eb2fe768-6e2a-4c37-a10b-3254bef1114b"). InnerVolumeSpecName "kube-api-access-ml4t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:11:40.095008 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.094974 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-kube-api-access-npr6f" (OuterVolumeSpecName: "kube-api-access-npr6f") pod "28e640c0-fd7b-40ec-8b0b-9432b3d536e7" (UID: "28e640c0-fd7b-40ec-8b0b-9432b3d536e7"). InnerVolumeSpecName "kube-api-access-npr6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:11:40.194538 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.194491 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2fe768-6e2a-4c37-a10b-3254bef1114b-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:11:40.194538 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.194530 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml4t2\" (UniqueName: \"kubernetes.io/projected/eb2fe768-6e2a-4c37-a10b-3254bef1114b-kube-api-access-ml4t2\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:11:40.194538 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.194540 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npr6f\" (UniqueName: \"kubernetes.io/projected/28e640c0-fd7b-40ec-8b0b-9432b3d536e7-kube-api-access-npr6f\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:11:40.239264 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.239230 2572 generic.go:356] "Generic (PLEG): container finished" podID="eb2fe768-6e2a-4c37-a10b-3254bef1114b" containerID="782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2" exitCode=0 Sep 10 05:11:40.239497 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.239280 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" Sep 10 05:11:40.239497 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.239318 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" event={"ID":"eb2fe768-6e2a-4c37-a10b-3254bef1114b","Type":"ContainerDied","Data":"782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2"} Sep 10 05:11:40.239497 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.239378 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh" event={"ID":"eb2fe768-6e2a-4c37-a10b-3254bef1114b","Type":"ContainerDied","Data":"244f0963172872f2f6152db98418a07898ccac4909ff164b07c553694e3cbb9f"} Sep 10 05:11:40.239497 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.239405 2572 scope.go:117] "RemoveContainer" containerID="782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2" Sep 10 05:11:40.240728 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.240710 2572 generic.go:356] "Generic (PLEG): container finished" podID="28e640c0-fd7b-40ec-8b0b-9432b3d536e7" containerID="f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691" exitCode=0 Sep 10 05:11:40.240805 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.240761 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" Sep 10 05:11:40.240805 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.240789 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" event={"ID":"28e640c0-fd7b-40ec-8b0b-9432b3d536e7","Type":"ContainerDied","Data":"f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691"} Sep 10 05:11:40.240932 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.240836 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld" event={"ID":"28e640c0-fd7b-40ec-8b0b-9432b3d536e7","Type":"ContainerDied","Data":"d2c5f09f3187b858622a350ee8057d2c3473ad06fc01d326e04b29ee1b857016"} Sep 10 05:11:40.249213 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.249190 2572 scope.go:117] "RemoveContainer" containerID="782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2" Sep 10 05:11:40.249568 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:11:40.249548 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2\": container with ID starting with 782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2 not found: ID does not exist" containerID="782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2" Sep 10 05:11:40.249638 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.249577 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2"} err="failed to get container status \"782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2\": rpc error: code = NotFound desc = could not find container \"782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2\": container with ID starting with 782fa7213a31cc0481a69802167fd8cd222c1ef8cbd617c0a4da3f1b8562b1a2 not found: ID does not exist" Sep 10 05:11:40.249638 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.249595 2572 scope.go:117] "RemoveContainer" containerID="f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691" Sep 10 05:11:40.258276 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.258256 2572 scope.go:117] "RemoveContainer" containerID="f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691" Sep 10 05:11:40.258688 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:11:40.258661 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691\": container with ID starting with f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691 not found: ID does not exist" containerID="f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691" Sep 10 05:11:40.258760 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.258698 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691"} err="failed to get container status \"f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691\": rpc error: code = NotFound desc = could not find container \"f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691\": container with ID starting with f62b0d0dfb5fdb0e2f22b11faba3fdb34e4a6575f2395b83771e9cbc41489691 not found: ID does not exist" Sep 10 05:11:40.265257 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.265222 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh"] Sep 10 05:11:40.269314 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.269282 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c7644-7f49987b6f-9g4xh"] Sep 10 05:11:40.280855 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.280821 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld"] Sep 10 05:11:40.284476 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:40.284432 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1c7644-58d9fdc9f8-fbkld"] Sep 10 05:11:41.037366 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:41.037332 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28e640c0-fd7b-40ec-8b0b-9432b3d536e7" path="/var/lib/kubelet/pods/28e640c0-fd7b-40ec-8b0b-9432b3d536e7/volumes" Sep 10 05:11:41.037764 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:41.037732 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb2fe768-6e2a-4c37-a10b-3254bef1114b" path="/var/lib/kubelet/pods/eb2fe768-6e2a-4c37-a10b-3254bef1114b/volumes" Sep 10 05:11:57.028885 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.028819 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7"] Sep 10 05:11:57.029586 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.029278 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" podUID="0c931a6d-1b88-417d-856f-43a1c3526a05" containerName="maas-api" containerID="cri-o://e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9" gracePeriod=30 Sep 10 05:11:57.090457 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:11:57.090398 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c931a6d_1b88_417d_856f_43a1c3526a05.slice/crio-e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c931a6d_1b88_417d_856f_43a1c3526a05.slice/crio-conmon-e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9.scope\": RecentStats: unable to find data in memory cache]" Sep 10 05:11:57.290596 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.290523 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:11:57.341506 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.341470 2572 generic.go:356] "Generic (PLEG): container finished" podID="0c931a6d-1b88-417d-856f-43a1c3526a05" containerID="e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9" exitCode=0 Sep 10 05:11:57.341717 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.341531 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" event={"ID":"0c931a6d-1b88-417d-856f-43a1c3526a05","Type":"ContainerDied","Data":"e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9"} Sep 10 05:11:57.341717 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.341543 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" Sep 10 05:11:57.341717 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.341594 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7" event={"ID":"0c931a6d-1b88-417d-856f-43a1c3526a05","Type":"ContainerDied","Data":"eb7f9b7ec826803ad49caf5154075236e8ac2b3995fd3ffed173ce6e1f25cb88"} Sep 10 05:11:57.341717 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.341625 2572 scope.go:117] "RemoveContainer" containerID="e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9" Sep 10 05:11:57.350996 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.350972 2572 scope.go:117] "RemoveContainer" containerID="e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9" Sep 10 05:11:57.351335 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:11:57.351315 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9\": container with ID starting with e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9 not found: ID does not exist" containerID="e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9" Sep 10 05:11:57.351411 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.351343 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9"} err="failed to get container status \"e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9\": rpc error: code = NotFound desc = could not find container \"e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9\": container with ID starting with e42a74b35df7161cb80b9da2de612d1608e147024eb97bc4b65d052d33855fe9 not found: ID does not exist" Sep 10 05:11:57.369305 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.369270 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-metrics-certs\") pod \"0c931a6d-1b88-417d-856f-43a1c3526a05\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " Sep 10 05:11:57.369518 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.369335 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-maas-api-tls\") pod \"0c931a6d-1b88-417d-856f-43a1c3526a05\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " Sep 10 05:11:57.369518 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.369399 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c931a6d-1b88-417d-856f-43a1c3526a05-openshift-service-ca\") pod \"0c931a6d-1b88-417d-856f-43a1c3526a05\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " Sep 10 05:11:57.369518 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.369492 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8mpq\" (UniqueName: \"kubernetes.io/projected/0c931a6d-1b88-417d-856f-43a1c3526a05-kube-api-access-k8mpq\") pod \"0c931a6d-1b88-417d-856f-43a1c3526a05\" (UID: \"0c931a6d-1b88-417d-856f-43a1c3526a05\") " Sep 10 05:11:57.369851 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.369816 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c931a6d-1b88-417d-856f-43a1c3526a05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c931a6d-1b88-417d-856f-43a1c3526a05" (UID: "0c931a6d-1b88-417d-856f-43a1c3526a05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:11:57.371591 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.371549 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0c931a6d-1b88-417d-856f-43a1c3526a05" (UID: "0c931a6d-1b88-417d-856f-43a1c3526a05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:11:57.371713 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.371603 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c931a6d-1b88-417d-856f-43a1c3526a05" (UID: "0c931a6d-1b88-417d-856f-43a1c3526a05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:11:57.371713 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.371669 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c931a6d-1b88-417d-856f-43a1c3526a05-kube-api-access-k8mpq" (OuterVolumeSpecName: "kube-api-access-k8mpq") pod "0c931a6d-1b88-417d-856f-43a1c3526a05" (UID: "0c931a6d-1b88-417d-856f-43a1c3526a05"). InnerVolumeSpecName "kube-api-access-k8mpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:11:57.471306 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.471261 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c931a6d-1b88-417d-856f-43a1c3526a05-openshift-service-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:11:57.471306 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.471302 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8mpq\" (UniqueName: \"kubernetes.io/projected/0c931a6d-1b88-417d-856f-43a1c3526a05-kube-api-access-k8mpq\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:11:57.471306 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.471313 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:11:57.471597 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.471323 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c931a6d-1b88-417d-856f-43a1c3526a05-maas-api-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:11:57.663656 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.663618 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7"] Sep 10 05:11:57.667671 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:57.667633 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4bfef6-64575cb6d8-462c7"] Sep 10 05:11:59.037191 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:11:59.037154 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c931a6d-1b88-417d-856f-43a1c3526a05" path="/var/lib/kubelet/pods/0c931a6d-1b88-417d-856f-43a1c3526a05/volumes" Sep 10 05:12:14.647901 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:14.647867 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q"] Sep 10 05:12:14.648506 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:14.648161 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" podUID="be95fba4-fccd-40a1-ad3f-ae114d98f754" containerName="payload-processing" containerID="cri-o://d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077" gracePeriod=30 Sep 10 05:12:14.922680 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:14.922651 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:12:14.944871 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:14.944837 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2qrk\" (UniqueName: \"kubernetes.io/projected/be95fba4-fccd-40a1-ad3f-ae114d98f754-kube-api-access-n2qrk\") pod \"be95fba4-fccd-40a1-ad3f-ae114d98f754\" (UID: \"be95fba4-fccd-40a1-ad3f-ae114d98f754\") " Sep 10 05:12:14.945046 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:14.944886 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be95fba4-fccd-40a1-ad3f-ae114d98f754-plugins-config-volume\") pod \"be95fba4-fccd-40a1-ad3f-ae114d98f754\" (UID: \"be95fba4-fccd-40a1-ad3f-ae114d98f754\") " Sep 10 05:12:14.945334 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:14.945293 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be95fba4-fccd-40a1-ad3f-ae114d98f754-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be95fba4-fccd-40a1-ad3f-ae114d98f754" (UID: "be95fba4-fccd-40a1-ad3f-ae114d98f754"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:12:14.947066 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:14.947035 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be95fba4-fccd-40a1-ad3f-ae114d98f754-kube-api-access-n2qrk" (OuterVolumeSpecName: "kube-api-access-n2qrk") pod "be95fba4-fccd-40a1-ad3f-ae114d98f754" (UID: "be95fba4-fccd-40a1-ad3f-ae114d98f754"). InnerVolumeSpecName "kube-api-access-n2qrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:12:15.046105 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:15.046058 2572 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be95fba4-fccd-40a1-ad3f-ae114d98f754-plugins-config-volume\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:12:15.046105 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:15.046106 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qrk\" (UniqueName: \"kubernetes.io/projected/be95fba4-fccd-40a1-ad3f-ae114d98f754-kube-api-access-n2qrk\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:12:15.437426 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:15.437390 2572 generic.go:356] "Generic (PLEG): container finished" podID="be95fba4-fccd-40a1-ad3f-ae114d98f754" containerID="d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077" exitCode=0 Sep 10 05:12:15.437635 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:15.437471 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" Sep 10 05:12:15.437635 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:15.437479 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" event={"ID":"be95fba4-fccd-40a1-ad3f-ae114d98f754","Type":"ContainerDied","Data":"d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077"} Sep 10 05:12:15.437635 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:15.437532 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q" event={"ID":"be95fba4-fccd-40a1-ad3f-ae114d98f754","Type":"ContainerDied","Data":"b196dd0faab0d9593ce4e90f838d41c7bead274885cd80498fcbccf8d5a397b5"} Sep 10 05:12:15.437635 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:15.437555 2572 scope.go:117] "RemoveContainer" containerID="d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077" Sep 10 05:12:15.447071 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:15.447051 2572 scope.go:117] "RemoveContainer" containerID="d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077" Sep 10 05:12:15.447425 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:12:15.447394 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077\": container with ID starting with d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077 not found: ID does not exist" containerID="d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077" Sep 10 05:12:15.447495 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:15.447434 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077"} err="failed to get container status \"d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077\": rpc error: code = NotFound desc = could not find container \"d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077\": container with ID starting with d7043532f5cf763e1c390338cacdb191d94701a3add0ca9313cfd7bea0160077 not found: ID does not exist" Sep 10 05:12:15.455241 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:15.455199 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q"] Sep 10 05:12:15.457130 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:15.457103 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3b702d-b9dc5c997-7sf2q"] Sep 10 05:12:17.037375 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:17.037341 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be95fba4-fccd-40a1-ad3f-ae114d98f754" path="/var/lib/kubelet/pods/be95fba4-fccd-40a1-ad3f-ae114d98f754/volumes" Sep 10 05:12:21.444171 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:21.444136 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46"] Sep 10 05:12:21.444623 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:21.444349 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" containerID="cri-o://6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830" gracePeriod=30 Sep 10 05:12:26.706890 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.706862 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:12:26.764066 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764029 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29deda9f-e309-4a78-a24b-465dd5b2ce00-istiod-ca-cert\") pod \"29deda9f-e309-4a78-a24b-465dd5b2ce00\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " Sep 10 05:12:26.764253 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764099 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-credential-socket\") pod \"29deda9f-e309-4a78-a24b-465dd5b2ce00\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " Sep 10 05:12:26.764253 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764167 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-envoy\") pod \"29deda9f-e309-4a78-a24b-465dd5b2ce00\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " Sep 10 05:12:26.764356 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764263 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-data\") pod \"29deda9f-e309-4a78-a24b-465dd5b2ce00\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " Sep 10 05:12:26.764356 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764306 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxp9v\" (UniqueName: \"kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-kube-api-access-jxp9v\") pod \"29deda9f-e309-4a78-a24b-465dd5b2ce00\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " Sep 10 05:12:26.764356 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764334 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-certs\") pod \"29deda9f-e309-4a78-a24b-465dd5b2ce00\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " Sep 10 05:12:26.764549 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764382 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-token\") pod \"29deda9f-e309-4a78-a24b-465dd5b2ce00\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " Sep 10 05:12:26.764549 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764412 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-socket\") pod \"29deda9f-e309-4a78-a24b-465dd5b2ce00\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " Sep 10 05:12:26.764549 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764498 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-podinfo\") pod \"29deda9f-e309-4a78-a24b-465dd5b2ce00\" (UID: \"29deda9f-e309-4a78-a24b-465dd5b2ce00\") " Sep 10 05:12:26.764549 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764516 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29deda9f-e309-4a78-a24b-465dd5b2ce00" (UID: "29deda9f-e309-4a78-a24b-465dd5b2ce00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:12:26.764738 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764576 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-data" (OuterVolumeSpecName: "istio-data") pod "29deda9f-e309-4a78-a24b-465dd5b2ce00" (UID: "29deda9f-e309-4a78-a24b-465dd5b2ce00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:12:26.764738 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764599 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29deda9f-e309-4a78-a24b-465dd5b2ce00-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29deda9f-e309-4a78-a24b-465dd5b2ce00" (UID: "29deda9f-e309-4a78-a24b-465dd5b2ce00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:12:26.764818 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764732 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29deda9f-e309-4a78-a24b-465dd5b2ce00" (UID: "29deda9f-e309-4a78-a24b-465dd5b2ce00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:12:26.764912 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764894 2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29deda9f-e309-4a78-a24b-465dd5b2ce00-istiod-ca-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:12:26.764952 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764920 2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-credential-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:12:26.764952 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764934 2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-data\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:12:26.764952 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.764948 2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:12:26.765193 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.765160 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29deda9f-e309-4a78-a24b-465dd5b2ce00" (UID: "29deda9f-e309-4a78-a24b-465dd5b2ce00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:12:26.766761 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.766723 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-kube-api-access-jxp9v" (OuterVolumeSpecName: "kube-api-access-jxp9v") pod "29deda9f-e309-4a78-a24b-465dd5b2ce00" (UID: "29deda9f-e309-4a78-a24b-465dd5b2ce00"). InnerVolumeSpecName "kube-api-access-jxp9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:12:26.767021 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.766993 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29deda9f-e309-4a78-a24b-465dd5b2ce00" (UID: "29deda9f-e309-4a78-a24b-465dd5b2ce00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:12:26.767138 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.767075 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29deda9f-e309-4a78-a24b-465dd5b2ce00" (UID: "29deda9f-e309-4a78-a24b-465dd5b2ce00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 05:12:26.767312 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.767293 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-token" (OuterVolumeSpecName: "istio-token") pod "29deda9f-e309-4a78-a24b-465dd5b2ce00" (UID: "29deda9f-e309-4a78-a24b-465dd5b2ce00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:12:26.866583 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.866537 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxp9v\" (UniqueName: \"kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-kube-api-access-jxp9v\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:12:26.866583 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.866582 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-token\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:12:26.866774 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.866603 2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-workload-socket\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:12:26.866774 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.866618 2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-podinfo\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:12:26.866774 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:26.866632 2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29deda9f-e309-4a78-a24b-465dd5b2ce00-istio-envoy\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:12:27.502193 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.502159 2572 generic.go:356] "Generic (PLEG): container finished" podID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerID="6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830" exitCode=0 Sep 10 05:12:27.502402 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.502212 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" event={"ID":"29deda9f-e309-4a78-a24b-465dd5b2ce00","Type":"ContainerDied","Data":"6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830"} Sep 10 05:12:27.502402 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.502218 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" Sep 10 05:12:27.502402 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.502256 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46" event={"ID":"29deda9f-e309-4a78-a24b-465dd5b2ce00","Type":"ContainerDied","Data":"4110027f9e8d6a7e512dfef620915922df91ffb0278438add1f0e347d6253033"} Sep 10 05:12:27.502402 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.502278 2572 scope.go:117] "RemoveContainer" containerID="6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830" Sep 10 05:12:27.511433 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.511413 2572 scope.go:117] "RemoveContainer" containerID="d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40" Sep 10 05:12:27.521090 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.521011 2572 scope.go:117] "RemoveContainer" containerID="6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830" Sep 10 05:12:27.521782 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.521758 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46"] Sep 10 05:12:27.521936 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:12:27.521912 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830\": container with ID starting with 6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830 not found: ID does not exist" containerID="6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830" Sep 10 05:12:27.522009 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.521949 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830"} err="failed to get container status \"6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830\": rpc error: code = NotFound desc = could not find container \"6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830\": container with ID starting with 6d78bf3aa3a2d4a45a6bd3f80a35d92fc37e981eb9e846ceee54f633672d4830 not found: ID does not exist" Sep 10 05:12:27.522009 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.521975 2572 scope.go:117] "RemoveContainer" containerID="d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40" Sep 10 05:12:27.522298 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:12:27.522278 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40\": container with ID starting with d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40 not found: ID does not exist" containerID="d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40" Sep 10 05:12:27.522357 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.522304 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40"} err="failed to get container status \"d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40\": rpc error: code = NotFound desc = could not find container \"d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40\": container with ID starting with d68e7b9359111e21577b6b3099b77e97a7df740c983744b4920df8473e66bc40 not found: ID does not exist" Sep 10 05:12:27.524000 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:27.523968 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3b702d-openshift-default-9d49444b6-j4z46"] Sep 10 05:12:29.037658 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:12:29.037622 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" path="/var/lib/kubelet/pods/29deda9f-e309-4a78-a24b-465dd5b2ce00/volumes" Sep 10 05:13:25.808501 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808462 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7597cb784c-x2zqf"] Sep 10 05:13:25.808909 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808881 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" Sep 10 05:13:25.808909 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808896 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" Sep 10 05:13:25.808909 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808907 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808914 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808922 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb2fe768-6e2a-4c37-a10b-3254bef1114b" containerName="payload-processing" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808928 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb2fe768-6e2a-4c37-a10b-3254bef1114b" containerName="payload-processing" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808938 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f3e7af-c185-4853-a967-df7c5e752669" containerName="maas-api" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808944 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f3e7af-c185-4853-a967-df7c5e752669" containerName="maas-api" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808952 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efbd9b4e-da25-4fcf-8d69-c45d1c879767" containerName="payload-pre-processing" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808957 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="efbd9b4e-da25-4fcf-8d69-c45d1c879767" containerName="payload-pre-processing" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808964 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e640c0-fd7b-40ec-8b0b-9432b3d536e7" containerName="payload-pre-processing" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808969 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e640c0-fd7b-40ec-8b0b-9432b3d536e7" containerName="payload-pre-processing" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808976 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808981 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808992 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec36820-a2a3-4cdd-8c09-0c3ce792361e" containerName="payload-pre-processing" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.808997 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec36820-a2a3-4cdd-8c09-0c3ce792361e" containerName="payload-pre-processing" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809003 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c931a6d-1b88-417d-856f-43a1c3526a05" containerName="maas-api" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809008 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c931a6d-1b88-417d-856f-43a1c3526a05" containerName="maas-api" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809013 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be95fba4-fccd-40a1-ad3f-ae114d98f754" containerName="payload-processing" Sep 10 05:13:25.809015 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809017 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="be95fba4-fccd-40a1-ad3f-ae114d98f754" containerName="payload-processing" Sep 10 05:13:25.809551 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809082 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="efbd9b4e-da25-4fcf-8d69-c45d1c879767" containerName="payload-pre-processing" Sep 10 05:13:25.809551 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809092 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f3e7af-c185-4853-a967-df7c5e752669" containerName="maas-api" Sep 10 05:13:25.809551 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809100 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c931a6d-1b88-417d-856f-43a1c3526a05" containerName="maas-api" Sep 10 05:13:25.809551 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809105 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" Sep 10 05:13:25.809551 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809110 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="28e640c0-fd7b-40ec-8b0b-9432b3d536e7" containerName="payload-pre-processing" Sep 10 05:13:25.809551 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809117 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec36820-a2a3-4cdd-8c09-0c3ce792361e" containerName="payload-pre-processing" Sep 10 05:13:25.809551 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809123 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" Sep 10 05:13:25.809551 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809129 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="be95fba4-fccd-40a1-ad3f-ae114d98f754" containerName="payload-processing" Sep 10 05:13:25.809551 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809135 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8570dd9c-7dec-4a4c-8a17-d5e33393c928" containerName="istio-proxy" Sep 10 05:13:25.809551 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809142 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" Sep 10 05:13:25.809551 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.809147 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb2fe768-6e2a-4c37-a10b-3254bef1114b" containerName="payload-processing" Sep 10 05:13:25.812113 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.812095 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:25.814758 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.814737 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 05:13:25.815878 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.815864 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 05:13:25.815945 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.815885 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z9tjc\"" Sep 10 05:13:25.815997 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.815862 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 05:13:25.832318 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.832186 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7597cb784c-x2zqf"] Sep 10 05:13:25.934028 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.933977 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-webhook-cert\") pod \"maas-controller-7597cb784c-x2zqf\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:25.934233 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.934074 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-metrics-certs\") pod \"maas-controller-7597cb784c-x2zqf\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:25.934233 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:25.934115 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwq9j\" (UniqueName: \"kubernetes.io/projected/bca2df1a-d0f1-45ba-b463-722435afd73f-kube-api-access-wwq9j\") pod \"maas-controller-7597cb784c-x2zqf\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:26.034846 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:26.034799 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-metrics-certs\") pod \"maas-controller-7597cb784c-x2zqf\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:26.034846 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:26.034849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwq9j\" (UniqueName: \"kubernetes.io/projected/bca2df1a-d0f1-45ba-b463-722435afd73f-kube-api-access-wwq9j\") pod \"maas-controller-7597cb784c-x2zqf\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:26.035070 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:26.034916 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-webhook-cert\") pod \"maas-controller-7597cb784c-x2zqf\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:26.037392 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:26.037349 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-webhook-cert\") pod \"maas-controller-7597cb784c-x2zqf\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:26.037392 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:26.037349 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-metrics-certs\") pod \"maas-controller-7597cb784c-x2zqf\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:26.042873 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:26.042841 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwq9j\" (UniqueName: \"kubernetes.io/projected/bca2df1a-d0f1-45ba-b463-722435afd73f-kube-api-access-wwq9j\") pod \"maas-controller-7597cb784c-x2zqf\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:26.126383 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:26.126287 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:26.401634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:26.401606 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7597cb784c-x2zqf"] Sep 10 05:13:26.404208 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:13:26.404172 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca2df1a_d0f1_45ba_b463_722435afd73f.slice/crio-e419ce995a511d7f00069b64f5fe4d1abe88cca23abc504f9c577ad56f0b1316 WatchSource:0}: Error finding container e419ce995a511d7f00069b64f5fe4d1abe88cca23abc504f9c577ad56f0b1316: Status 404 returned error can't find the container with id e419ce995a511d7f00069b64f5fe4d1abe88cca23abc504f9c577ad56f0b1316 Sep 10 05:13:26.405924 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:26.405896 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 05:13:26.797289 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:26.797252 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7597cb784c-x2zqf" event={"ID":"bca2df1a-d0f1-45ba-b463-722435afd73f","Type":"ContainerStarted","Data":"e419ce995a511d7f00069b64f5fe4d1abe88cca23abc504f9c577ad56f0b1316"} Sep 10 05:13:27.804396 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:27.804354 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7597cb784c-x2zqf" event={"ID":"bca2df1a-d0f1-45ba-b463-722435afd73f","Type":"ContainerStarted","Data":"0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243"} Sep 10 05:13:27.804803 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:27.804424 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:27.821674 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:27.821620 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7597cb784c-x2zqf" podStartSLOduration=2.425501693 podStartE2EDuration="2.821603002s" totalImagesPullingTime="396.101309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:13:27.819385067 +0000 UTC m=+1927.323218139" watchObservedRunningTime="2026-09-10 05:13:27.821603002 +0000 UTC m=+1927.325436113" Sep 10 05:13:38.812361 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:38.812325 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:46.097751 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.097713 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7597cb784c-x2zqf"] Sep 10 05:13:46.098267 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.097932 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7597cb784c-x2zqf" podUID="bca2df1a-d0f1-45ba-b463-722435afd73f" containerName="manager" containerID="cri-o://0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243" gracePeriod=30 Sep 10 05:13:46.347259 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.347233 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:46.423836 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.423744 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwq9j\" (UniqueName: \"kubernetes.io/projected/bca2df1a-d0f1-45ba-b463-722435afd73f-kube-api-access-wwq9j\") pod \"bca2df1a-d0f1-45ba-b463-722435afd73f\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " Sep 10 05:13:46.423836 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.423796 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-metrics-certs\") pod \"bca2df1a-d0f1-45ba-b463-722435afd73f\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " Sep 10 05:13:46.423836 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.423828 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-webhook-cert\") pod \"bca2df1a-d0f1-45ba-b463-722435afd73f\" (UID: \"bca2df1a-d0f1-45ba-b463-722435afd73f\") " Sep 10 05:13:46.426256 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.426224 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bca2df1a-d0f1-45ba-b463-722435afd73f" (UID: "bca2df1a-d0f1-45ba-b463-722435afd73f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:13:46.426406 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.426330 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bca2df1a-d0f1-45ba-b463-722435afd73f" (UID: "bca2df1a-d0f1-45ba-b463-722435afd73f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:13:46.426406 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.426339 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca2df1a-d0f1-45ba-b463-722435afd73f-kube-api-access-wwq9j" (OuterVolumeSpecName: "kube-api-access-wwq9j") pod "bca2df1a-d0f1-45ba-b463-722435afd73f" (UID: "bca2df1a-d0f1-45ba-b463-722435afd73f"). InnerVolumeSpecName "kube-api-access-wwq9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:13:46.525305 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.525264 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwq9j\" (UniqueName: \"kubernetes.io/projected/bca2df1a-d0f1-45ba-b463-722435afd73f-kube-api-access-wwq9j\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:13:46.525305 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.525297 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:13:46.525305 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.525307 2572 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bca2df1a-d0f1-45ba-b463-722435afd73f-webhook-cert\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:13:46.906888 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.906847 2572 generic.go:356] "Generic (PLEG): container finished" podID="bca2df1a-d0f1-45ba-b463-722435afd73f" containerID="0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243" exitCode=0 Sep 10 05:13:46.906888 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.906891 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7597cb784c-x2zqf" Sep 10 05:13:46.907156 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.906930 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7597cb784c-x2zqf" event={"ID":"bca2df1a-d0f1-45ba-b463-722435afd73f","Type":"ContainerDied","Data":"0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243"} Sep 10 05:13:46.907156 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.906982 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7597cb784c-x2zqf" event={"ID":"bca2df1a-d0f1-45ba-b463-722435afd73f","Type":"ContainerDied","Data":"e419ce995a511d7f00069b64f5fe4d1abe88cca23abc504f9c577ad56f0b1316"} Sep 10 05:13:46.907156 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.907003 2572 scope.go:117] "RemoveContainer" containerID="0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243" Sep 10 05:13:46.916258 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.916239 2572 scope.go:117] "RemoveContainer" containerID="0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243" Sep 10 05:13:46.916577 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:13:46.916559 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243\": container with ID starting with 0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243 not found: ID does not exist" containerID="0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243" Sep 10 05:13:46.916646 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.916604 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243"} err="failed to get container status \"0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243\": rpc error: code = NotFound desc = could not find container \"0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243\": container with ID starting with 0e8c42d7fb95661106bf5f1cbefd170b070f3e59577300240576efc1fabf8243 not found: ID does not exist" Sep 10 05:13:46.927907 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.927882 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7597cb784c-x2zqf"] Sep 10 05:13:46.931276 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:46.931249 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7597cb784c-x2zqf"] Sep 10 05:13:47.036960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:47.036928 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bca2df1a-d0f1-45ba-b463-722435afd73f" path="/var/lib/kubelet/pods/bca2df1a-d0f1-45ba-b463-722435afd73f/volumes" Sep 10 05:13:48.087303 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.087265 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7597cb784c-nvv8r"] Sep 10 05:13:48.087734 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.087669 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca2df1a-d0f1-45ba-b463-722435afd73f" containerName="manager" Sep 10 05:13:48.087734 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.087680 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca2df1a-d0f1-45ba-b463-722435afd73f" containerName="manager" Sep 10 05:13:48.087734 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.087692 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" Sep 10 05:13:48.087734 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.087698 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="29deda9f-e309-4a78-a24b-465dd5b2ce00" containerName="istio-proxy" Sep 10 05:13:48.087931 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.087752 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca2df1a-d0f1-45ba-b463-722435afd73f" containerName="manager" Sep 10 05:13:48.092373 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.092354 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:48.095821 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.095797 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z9tjc\"" Sep 10 05:13:48.095821 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.095810 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 05:13:48.096020 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.095808 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 05:13:48.096020 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.095877 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 05:13:48.099640 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.099614 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7597cb784c-nvv8r"] Sep 10 05:13:48.139586 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.139550 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrp9d\" (UniqueName: \"kubernetes.io/projected/a759187e-0dfc-4b81-8b64-a8c657aba792-kube-api-access-lrp9d\") pod \"maas-controller-7597cb784c-nvv8r\" (UID: \"a759187e-0dfc-4b81-8b64-a8c657aba792\") " pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:48.139770 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.139626 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a759187e-0dfc-4b81-8b64-a8c657aba792-webhook-cert\") pod \"maas-controller-7597cb784c-nvv8r\" (UID: \"a759187e-0dfc-4b81-8b64-a8c657aba792\") " pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:48.139770 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.139653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a759187e-0dfc-4b81-8b64-a8c657aba792-metrics-certs\") pod \"maas-controller-7597cb784c-nvv8r\" (UID: \"a759187e-0dfc-4b81-8b64-a8c657aba792\") " pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:48.240679 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.240639 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a759187e-0dfc-4b81-8b64-a8c657aba792-webhook-cert\") pod \"maas-controller-7597cb784c-nvv8r\" (UID: \"a759187e-0dfc-4b81-8b64-a8c657aba792\") " pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:48.240679 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.240682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a759187e-0dfc-4b81-8b64-a8c657aba792-metrics-certs\") pod \"maas-controller-7597cb784c-nvv8r\" (UID: \"a759187e-0dfc-4b81-8b64-a8c657aba792\") " pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:48.240934 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.240756 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrp9d\" (UniqueName: \"kubernetes.io/projected/a759187e-0dfc-4b81-8b64-a8c657aba792-kube-api-access-lrp9d\") pod \"maas-controller-7597cb784c-nvv8r\" (UID: \"a759187e-0dfc-4b81-8b64-a8c657aba792\") " pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:48.243236 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.243208 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a759187e-0dfc-4b81-8b64-a8c657aba792-webhook-cert\") pod \"maas-controller-7597cb784c-nvv8r\" (UID: \"a759187e-0dfc-4b81-8b64-a8c657aba792\") " pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:48.243350 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.243266 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a759187e-0dfc-4b81-8b64-a8c657aba792-metrics-certs\") pod \"maas-controller-7597cb784c-nvv8r\" (UID: \"a759187e-0dfc-4b81-8b64-a8c657aba792\") " pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:48.248583 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.248557 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrp9d\" (UniqueName: \"kubernetes.io/projected/a759187e-0dfc-4b81-8b64-a8c657aba792-kube-api-access-lrp9d\") pod \"maas-controller-7597cb784c-nvv8r\" (UID: \"a759187e-0dfc-4b81-8b64-a8c657aba792\") " pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:48.405061 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.404967 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:48.678579 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.678335 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7597cb784c-nvv8r"] Sep 10 05:13:48.681358 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:13:48.681325 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda759187e_0dfc_4b81_8b64_a8c657aba792.slice/crio-9b68f23d9d414e2eff25e3068ef04974dcf1fbb814fff3c4e09209a1e07aa80a WatchSource:0}: Error finding container 9b68f23d9d414e2eff25e3068ef04974dcf1fbb814fff3c4e09209a1e07aa80a: Status 404 returned error can't find the container with id 9b68f23d9d414e2eff25e3068ef04974dcf1fbb814fff3c4e09209a1e07aa80a Sep 10 05:13:48.918850 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:48.918810 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7597cb784c-nvv8r" event={"ID":"a759187e-0dfc-4b81-8b64-a8c657aba792","Type":"ContainerStarted","Data":"9b68f23d9d414e2eff25e3068ef04974dcf1fbb814fff3c4e09209a1e07aa80a"} Sep 10 05:13:49.924753 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:49.924717 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7597cb784c-nvv8r" event={"ID":"a759187e-0dfc-4b81-8b64-a8c657aba792","Type":"ContainerStarted","Data":"0078ba6ea36cb8cb258ddcf6c69484046d4bddf1e0467eb4bf62f23c2b2b9385"} Sep 10 05:13:49.925121 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:49.924845 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:13:49.940645 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:13:49.940596 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7597cb784c-nvv8r" podStartSLOduration=1.4818257479999999 podStartE2EDuration="1.940580372s" totalImagesPullingTime="458.754624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:13:49.939213708 +0000 UTC m=+1949.443046802" watchObservedRunningTime="2026-09-10 05:13:49.940580372 +0000 UTC m=+1949.444413433" Sep 10 05:14:00.932530 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:14:00.932492 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7597cb784c-nvv8r" Sep 10 05:17:06.489219 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.489186 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl"] Sep 10 05:17:06.493181 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.493157 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.495839 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.495810 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4d897bf1-kserve-self-signed-certs\"" Sep 10 05:17:06.503796 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.503767 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl"] Sep 10 05:17:06.518193 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.518155 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-home\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.518427 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.518200 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-model-cache\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.518427 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.518233 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tmp-dir\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.518427 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.518317 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db9rk\" (UniqueName: \"kubernetes.io/projected/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-kube-api-access-db9rk\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.518612 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.518423 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-dshm\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.518612 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.518488 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tls-certs\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.619607 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.619564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db9rk\" (UniqueName: \"kubernetes.io/projected/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-kube-api-access-db9rk\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.619797 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.619656 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-dshm\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.619797 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.619680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tls-certs\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.619797 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.619725 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-home\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.619797 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.619747 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-model-cache\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.619983 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.619874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tmp-dir\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.620162 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.620133 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-model-cache\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.620247 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.620189 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-home\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.620247 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.620225 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tmp-dir\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.622212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.622182 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-dshm\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.622410 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.622392 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tls-certs\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.626891 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.626867 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db9rk\" (UniqueName: \"kubernetes.io/projected/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-kube-api-access-db9rk\") pod \"e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:06.804233 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:06.804141 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:07.076989 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:07.076962 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl"] Sep 10 05:17:07.079003 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:17:07.078973 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod274f0ed2_54b7_4391_8f1e_7a5ab80af8fe.slice/crio-314cec68c65da30e839155f79492279c6d7df1dd864374241f964c1aa4867519 WatchSource:0}: Error finding container 314cec68c65da30e839155f79492279c6d7df1dd864374241f964c1aa4867519: Status 404 returned error can't find the container with id 314cec68c65da30e839155f79492279c6d7df1dd864374241f964c1aa4867519 Sep 10 05:17:07.934601 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:07.934560 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" event={"ID":"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe","Type":"ContainerStarted","Data":"c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11"} Sep 10 05:17:07.934601 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:07.934606 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" event={"ID":"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe","Type":"ContainerStarted","Data":"314cec68c65da30e839155f79492279c6d7df1dd864374241f964c1aa4867519"} Sep 10 05:17:07.951637 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:07.951578 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" podStartSLOduration=1.951561012 podStartE2EDuration="1.951561012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:17:07.950248836 +0000 UTC m=+2147.454081897" watchObservedRunningTime="2026-09-10 05:17:07.951561012 +0000 UTC m=+2147.455394071" Sep 10 05:17:16.805136 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:16.805038 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:16.805136 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:16.805100 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:16.817689 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:16.817659 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:16.990943 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:16.990904 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:17:46.694760 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:17:46.694712 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl"] Sep 10 05:18:01.716650 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:01.716584 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" podUID="274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" containerName="main" containerID="cri-o://c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11" gracePeriod=45 Sep 10 05:18:01.966950 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:01.966881 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:18:02.074096 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074049 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-dshm\") pod \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " Sep 10 05:18:02.074096 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074105 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tmp-dir\") pod \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " Sep 10 05:18:02.074367 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074147 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-model-cache\") pod \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " Sep 10 05:18:02.074367 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074174 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db9rk\" (UniqueName: \"kubernetes.io/projected/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-kube-api-access-db9rk\") pod \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " Sep 10 05:18:02.074367 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074203 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-home\") pod \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " Sep 10 05:18:02.074367 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074239 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tls-certs\") pod \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\" (UID: \"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe\") " Sep 10 05:18:02.074621 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074489 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" (UID: "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:18:02.074621 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074506 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-home" (OuterVolumeSpecName: "home") pod "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" (UID: "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:18:02.074621 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074522 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" (UID: "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:18:02.074778 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074673 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-model-cache\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:18:02.074778 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074690 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-home\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:18:02.074778 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.074732 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tmp-dir\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:18:02.076534 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.076510 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-dshm" (OuterVolumeSpecName: "dshm") pod "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" (UID: "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 05:18:02.076534 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.076515 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" (UID: "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:18:02.076675 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.076611 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-kube-api-access-db9rk" (OuterVolumeSpecName: "kube-api-access-db9rk") pod "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" (UID: "274f0ed2-54b7-4391-8f1e-7a5ab80af8fe"). InnerVolumeSpecName "kube-api-access-db9rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:18:02.175769 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.175728 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-tls-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:18:02.175769 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.175761 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-dshm\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:18:02.175769 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.175773 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db9rk\" (UniqueName: \"kubernetes.io/projected/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe-kube-api-access-db9rk\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:18:02.202050 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.202012 2572 generic.go:356] "Generic (PLEG): container finished" podID="274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" containerID="c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11" exitCode=0 Sep 10 05:18:02.202219 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.202057 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" Sep 10 05:18:02.202219 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.202095 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" event={"ID":"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe","Type":"ContainerDied","Data":"c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11"} Sep 10 05:18:02.202219 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.202143 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl" event={"ID":"274f0ed2-54b7-4391-8f1e-7a5ab80af8fe","Type":"ContainerDied","Data":"314cec68c65da30e839155f79492279c6d7df1dd864374241f964c1aa4867519"} Sep 10 05:18:02.202219 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.202163 2572 scope.go:117] "RemoveContainer" containerID="c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11" Sep 10 05:18:02.211884 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.211864 2572 scope.go:117] "RemoveContainer" containerID="c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11" Sep 10 05:18:02.212179 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:18:02.212158 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11\": container with ID starting with c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11 not found: ID does not exist" containerID="c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11" Sep 10 05:18:02.212247 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.212191 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11"} err="failed to get container status \"c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11\": rpc error: code = NotFound desc = could not find container \"c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11\": container with ID starting with c4e7b2da0914bc56ea38bde704c7b8ea9f21837d70abc0e089fc4f0a5211ce11 not found: ID does not exist" Sep 10 05:18:02.223251 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.223169 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl"] Sep 10 05:18:02.227342 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:02.227314 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4d897bf1-kserve-8659cd4479-snmbl"] Sep 10 05:18:03.036802 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:03.036768 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" path="/var/lib/kubelet/pods/274f0ed2-54b7-4391-8f1e-7a5ab80af8fe/volumes" Sep 10 05:18:10.028390 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.028354 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f"] Sep 10 05:18:10.030962 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.029324 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" containerName="main" Sep 10 05:18:10.030962 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.029365 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" containerName="main" Sep 10 05:18:10.030962 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.029524 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="274f0ed2-54b7-4391-8f1e-7a5ab80af8fe" containerName="main" Sep 10 05:18:10.046120 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.046072 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f"] Sep 10 05:18:10.046570 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.046482 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.159386 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.159346 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8150d65a-b833-4af0-8127-2e0bae2cb8c6-metrics-certs\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.159386 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.159393 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8150d65a-b833-4af0-8127-2e0bae2cb8c6-openshift-service-ca\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.159691 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.159625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8150d65a-b833-4af0-8127-2e0bae2cb8c6-maas-api-tls\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.159750 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.159725 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66zdr\" (UniqueName: \"kubernetes.io/projected/8150d65a-b833-4af0-8127-2e0bae2cb8c6-kube-api-access-66zdr\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.261258 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.261214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8150d65a-b833-4af0-8127-2e0bae2cb8c6-openshift-service-ca\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.261518 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.261333 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8150d65a-b833-4af0-8127-2e0bae2cb8c6-maas-api-tls\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.261518 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.261381 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66zdr\" (UniqueName: \"kubernetes.io/projected/8150d65a-b833-4af0-8127-2e0bae2cb8c6-kube-api-access-66zdr\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.261518 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.261436 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8150d65a-b833-4af0-8127-2e0bae2cb8c6-metrics-certs\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.261911 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.261890 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8150d65a-b833-4af0-8127-2e0bae2cb8c6-openshift-service-ca\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.263905 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.263883 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8150d65a-b833-4af0-8127-2e0bae2cb8c6-metrics-certs\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.264019 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.263940 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8150d65a-b833-4af0-8127-2e0bae2cb8c6-maas-api-tls\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.270430 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.270398 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66zdr\" (UniqueName: \"kubernetes.io/projected/8150d65a-b833-4af0-8127-2e0bae2cb8c6-kube-api-access-66zdr\") pod \"maas-api-84bff48b9d-4m47f\" (UID: \"8150d65a-b833-4af0-8127-2e0bae2cb8c6\") " pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.367191 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.367092 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:10.644611 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:10.644504 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f"] Sep 10 05:18:10.648861 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:18:10.648829 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8150d65a_b833_4af0_8127_2e0bae2cb8c6.slice/crio-ad0817484c4ece78ddce4db5db40f87ea5ea2a872475648d88b2fe1af74d0177 WatchSource:0}: Error finding container ad0817484c4ece78ddce4db5db40f87ea5ea2a872475648d88b2fe1af74d0177: Status 404 returned error can't find the container with id ad0817484c4ece78ddce4db5db40f87ea5ea2a872475648d88b2fe1af74d0177 Sep 10 05:18:11.260005 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:11.259964 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" event={"ID":"8150d65a-b833-4af0-8127-2e0bae2cb8c6","Type":"ContainerStarted","Data":"7744e6708842759128f01da0a59c28594b2008b96d68a7f7b03ede01b008b61d"} Sep 10 05:18:11.260406 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:11.260026 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" event={"ID":"8150d65a-b833-4af0-8127-2e0bae2cb8c6","Type":"ContainerStarted","Data":"ad0817484c4ece78ddce4db5db40f87ea5ea2a872475648d88b2fe1af74d0177"} Sep 10 05:18:11.260406 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:11.260048 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:11.276388 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:11.276281 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" podStartSLOduration=0.913759116 podStartE2EDuration="1.276265196s" totalImagesPullingTime="362.50608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 05:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:18:11.275181398 +0000 UTC m=+2210.779014460" watchObservedRunningTime="2026-09-10 05:18:11.276265196 +0000 UTC m=+2210.780098256" Sep 10 05:18:17.271579 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:17.271544 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84bff48b9d-4m47f" Sep 10 05:18:25.012972 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:25.012932 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5"] Sep 10 05:18:25.013917 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:25.013196 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" podUID="a9557d6f-fdc3-4260-8863-cb628ebafa62" containerName="maas-api" containerID="cri-o://1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6" gracePeriod=30 Sep 10 05:18:26.265214 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.265182 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 05:18:26.300361 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.300326 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-metrics-certs\") pod \"a9557d6f-fdc3-4260-8863-cb628ebafa62\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " Sep 10 05:18:26.300579 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.300478 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vhfz\" (UniqueName: \"kubernetes.io/projected/a9557d6f-fdc3-4260-8863-cb628ebafa62-kube-api-access-6vhfz\") pod \"a9557d6f-fdc3-4260-8863-cb628ebafa62\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " Sep 10 05:18:26.300579 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.300530 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-maas-api-tls\") pod \"a9557d6f-fdc3-4260-8863-cb628ebafa62\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " Sep 10 05:18:26.300579 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.300569 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9557d6f-fdc3-4260-8863-cb628ebafa62-openshift-service-ca\") pod \"a9557d6f-fdc3-4260-8863-cb628ebafa62\" (UID: \"a9557d6f-fdc3-4260-8863-cb628ebafa62\") " Sep 10 05:18:26.300998 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.300957 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9557d6f-fdc3-4260-8863-cb628ebafa62-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a9557d6f-fdc3-4260-8863-cb628ebafa62" (UID: "a9557d6f-fdc3-4260-8863-cb628ebafa62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 05:18:26.302731 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.302702 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a9557d6f-fdc3-4260-8863-cb628ebafa62" (UID: "a9557d6f-fdc3-4260-8863-cb628ebafa62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:18:26.302843 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.302783 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9557d6f-fdc3-4260-8863-cb628ebafa62-kube-api-access-6vhfz" (OuterVolumeSpecName: "kube-api-access-6vhfz") pod "a9557d6f-fdc3-4260-8863-cb628ebafa62" (UID: "a9557d6f-fdc3-4260-8863-cb628ebafa62"). InnerVolumeSpecName "kube-api-access-6vhfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 05:18:26.302980 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.302963 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a9557d6f-fdc3-4260-8863-cb628ebafa62" (UID: "a9557d6f-fdc3-4260-8863-cb628ebafa62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 05:18:26.348905 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.348814 2572 generic.go:356] "Generic (PLEG): container finished" podID="a9557d6f-fdc3-4260-8863-cb628ebafa62" containerID="1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6" exitCode=0 Sep 10 05:18:26.348905 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.348859 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" Sep 10 05:18:26.348905 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.348883 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" event={"ID":"a9557d6f-fdc3-4260-8863-cb628ebafa62","Type":"ContainerDied","Data":"1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6"} Sep 10 05:18:26.349169 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.348925 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5" event={"ID":"a9557d6f-fdc3-4260-8863-cb628ebafa62","Type":"ContainerDied","Data":"445b0378c6466a4acf4bbb8582a0690301036ea21cab960a353bf618fbaf237f"} Sep 10 05:18:26.349169 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.348946 2572 scope.go:117] "RemoveContainer" containerID="1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6" Sep 10 05:18:26.358920 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.358898 2572 scope.go:117] "RemoveContainer" containerID="1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6" Sep 10 05:18:26.359240 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:18:26.359220 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6\": container with ID starting with 1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6 not found: ID does not exist" containerID="1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6" Sep 10 05:18:26.359289 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.359250 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6"} err="failed to get container status \"1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6\": rpc error: code = NotFound desc = could not find container \"1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6\": container with ID starting with 1727ee86f3ebdb729672dc8d80e3204c58c89fe181be8efcd89770d0fdb5fbd6 not found: ID does not exist" Sep 10 05:18:26.370805 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.370770 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5"] Sep 10 05:18:26.372572 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.372548 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff5bf8f9d-r47g5"] Sep 10 05:18:26.401499 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.401430 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vhfz\" (UniqueName: \"kubernetes.io/projected/a9557d6f-fdc3-4260-8863-cb628ebafa62-kube-api-access-6vhfz\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:18:26.401499 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.401497 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-maas-api-tls\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:18:26.401499 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.401510 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9557d6f-fdc3-4260-8863-cb628ebafa62-openshift-service-ca\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:18:26.401752 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:26.401518 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9557d6f-fdc3-4260-8863-cb628ebafa62-metrics-certs\") on node \"ip-10-0-136-213.ec2.internal\" DevicePath \"\"" Sep 10 05:18:27.037428 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:27.037383 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9557d6f-fdc3-4260-8863-cb628ebafa62" path="/var/lib/kubelet/pods/a9557d6f-fdc3-4260-8863-cb628ebafa62/volumes" Sep 10 05:18:46.374935 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:46.374822 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7597cb784c-nvv8r_a759187e-0dfc-4b81-8b64-a8c657aba792/manager/0.log" Sep 10 05:18:46.727230 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:46.727156 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c57f4866c-g8gh8_75eeed64-56c0-47e8-a25f-c61c067b69e3/copy-manifests/0.log" Sep 10 05:18:46.733565 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:46.733528 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c57f4866c-g8gh8_75eeed64-56c0-47e8-a25f-c61c067b69e3/manager/0.log" Sep 10 05:18:47.348857 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:47.348817 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84bff48b9d-4m47f_8150d65a-b833-4af0-8127-2e0bae2cb8c6/maas-api/0.log" Sep 10 05:18:49.624996 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:49.624948 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp_c37350ae-7be6-421a-a2ce-0ee47bacd479/istio-proxy/0.log" Sep 10 05:18:49.744762 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:49.744725 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dgxnt_fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b/discovery/0.log" Sep 10 05:18:50.087297 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:50.087262 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9l5xw_716fa502-5c07-487c-9410-4546ddfffec2/istio-proxy/1.log" Sep 10 05:18:50.336125 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:50.336093 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4nxnk_cf246a19-caf3-4fd6-8611-add57e445231/payload-processing/0.log" Sep 10 05:18:50.804920 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:50.804885 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-7w86s_6f215e50-9f97-4e49-aa61-a46a99ae38a6/main/0.log" Sep 10 05:18:59.977024 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:18:59.976971 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mh4x5_46bf0532-232f-423a-add2-7c0ff0a5a9a9/global-pull-secret-syncer/0.log" Sep 10 05:19:06.139887 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:06.139858 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f/alertmanager/0.log" Sep 10 05:19:08.440808 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:19:08.440757 2572 controller.go:195] "Failed to update lease" err="Put \"https://ac3232b46adf640648a846480a2e3a71-5c66a6bdc2c3d8a2.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-136-213.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 10 05:19:17.420430 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:17.420384 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp_c37350ae-7be6-421a-a2ce-0ee47bacd479/istio-proxy/0.log" Sep 10 05:19:17.891843 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:17.891804 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dgxnt_fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b/discovery/0.log" Sep 10 05:19:17.935116 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:17.935085 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9l5xw_716fa502-5c07-487c-9410-4546ddfffec2/istio-proxy/0.log" Sep 10 05:19:17.939925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:17.939894 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9l5xw_716fa502-5c07-487c-9410-4546ddfffec2/istio-proxy/1.log" Sep 10 05:19:17.998739 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:17.998708 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4nxnk_cf246a19-caf3-4fd6-8611-add57e445231/payload-processing/0.log" Sep 10 05:19:18.441130 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:19:18.441082 2572 controller.go:195] "Failed to update lease" err="Put \"https://ac3232b46adf640648a846480a2e3a71-5c66a6bdc2c3d8a2.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-136-213.ec2.internal?timeout=10s\": context deadline exceeded" Sep 10 05:19:19.189232 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:19.189194 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp_c37350ae-7be6-421a-a2ce-0ee47bacd479/istio-proxy/0.log" Sep 10 05:19:19.663567 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:19.663535 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dgxnt_fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b/discovery/0.log" Sep 10 05:19:19.710009 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:19.709951 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9l5xw_716fa502-5c07-487c-9410-4546ddfffec2/istio-proxy/0.log" Sep 10 05:19:19.715927 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:19.715896 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9l5xw_716fa502-5c07-487c-9410-4546ddfffec2/istio-proxy/1.log" Sep 10 05:19:19.776021 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:19.775987 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4nxnk_cf246a19-caf3-4fd6-8611-add57e445231/payload-processing/0.log" Sep 10 05:19:28.441740 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:19:28.441690 2572 controller.go:195] "Failed to update lease" err="Put \"https://ac3232b46adf640648a846480a2e3a71-5c66a6bdc2c3d8a2.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-136-213.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 10 05:19:38.442543 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:19:38.442485 2572 controller.go:195] "Failed to update lease" err="Put \"https://ac3232b46adf640648a846480a2e3a71-5c66a6bdc2c3d8a2.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-136-213.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 10 05:19:41.900466 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900411 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksvhk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900500 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900506 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gx84n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900514 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900522 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"default-dockercfg-vhk75\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900539 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hv8zz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900553 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900579 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900581 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900592 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900611 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d4254\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900642 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900654 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900678 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900713 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900722 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900732 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900801 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.900960 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900808 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900839 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4f2wj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900856 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900871 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jksxr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900893 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900928 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900963 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.900988 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901239 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901282 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6nh906m7lfq4a\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901323 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901360 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901395 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"default-dockercfg-756kd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901431 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901498 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dzbdb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901536 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v49sd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901139 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9hz5t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901604 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.901925 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901574 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nj7q4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901634 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-b9bc6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901667 2572 reflector.go:571] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901697 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901676 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901716 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7c7zc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901747 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901777 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901793 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901828 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901851 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901854 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901857 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901883 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901907 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901888 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901926 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-bstcg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901948 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kf5vp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.902868 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901953 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901977 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901991 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902015 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jxftf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902025 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902041 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902057 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902063 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902068 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901951 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902083 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902101 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.901993 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902153 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902118 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902111 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dhnvv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902201 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902213 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.903634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902228 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902249 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m66hn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902252 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdnb2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902263 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9hw97\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902280 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902286 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902305 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902307 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-t6jgh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902370 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902323 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j4jxp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902337 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s677c\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902357 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902393 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902409 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902418 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902431 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902437 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902476 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-b6stt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904212 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902487 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902377 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902509 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902518 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902522 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902536 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902539 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zzfzt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902548 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902568 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902584 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-khd29\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902592 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902478 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902487 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902614 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z9tjc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902617 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902332 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7k27c\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902635 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902281 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.904800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902511 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902655 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902666 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902674 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902689 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902695 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902508 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902717 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902728 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jbhx5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902364 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902743 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902751 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902755 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902636 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902772 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902696 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9k46b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902787 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902799 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.905400 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902805 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902767 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902779 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902829 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902836 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902569 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hwcjp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902730 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kghkv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902858 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902863 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902874 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902878 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902775 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-25t8d\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902892 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902910 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902914 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902716 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902895 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902855 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906048 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902739 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902948 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902885 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902935 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vhrd6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902918 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902947 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fi2tqmklkgpsj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: E0910 05:19:41.902863 2572 controller.go:195] "Failed to update lease" err="Put \"https://ac3232b46adf640648a846480a2e3a71-5c66a6bdc2c3d8a2.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-136-213.ec2.internal?timeout=10s\": http2: client connection lost" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902980 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902430 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cqf2m\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902991 2572 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902802 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902432 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902911 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902974 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902995 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-bifgalq23t6sr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:41.906634 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:41.902983 2572 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Sep 10 05:19:42.467118 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.467075 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f/config-reloader/0.log" Sep 10 05:19:42.489765 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.489732 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f/kube-rbac-proxy-web/0.log" Sep 10 05:19:42.510254 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.510226 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f/kube-rbac-proxy/0.log" Sep 10 05:19:42.530149 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.530104 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f/kube-rbac-proxy-metric/0.log" Sep 10 05:19:42.552387 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.552346 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f/prom-label-proxy/0.log" Sep 10 05:19:42.567434 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.567399 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ltxkf_c6759e6f-cc18-4a5e-bdbf-48aa0bdca13e/konnectivity-agent/0.log" Sep 10 05:19:42.576916 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.576888 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8a51d9fd-0ce4-4c0a-ac56-14f6b6d2537f/init-config-reloader/0.log" Sep 10 05:19:42.627339 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.627309 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-213.ec2.internal_ee62c14ad64bc1c5c73f7816f7b32bcc/haproxy/0.log" Sep 10 05:19:42.643797 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.643766 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6t7c5_400d2682-da56-4ff8-adc3-0d514bb7f1be/kube-state-metrics/0.log" Sep 10 05:19:42.659537 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.659506 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6t7c5_400d2682-da56-4ff8-adc3-0d514bb7f1be/kube-rbac-proxy-main/0.log" Sep 10 05:19:42.683289 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.683262 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-6t7c5_400d2682-da56-4ff8-adc3-0d514bb7f1be/kube-rbac-proxy-self/0.log" Sep 10 05:19:42.708923 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.708884 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hv8zz\"" Sep 10 05:19:42.709920 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.709903 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 05:19:42.712479 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.712423 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-77b8b9cc78-d2hfw_ea31ddbd-8d97-440e-a793-a56f5fc9ab12/metrics-server/0.log" Sep 10 05:19:42.713295 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.713279 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 05:19:42.715056 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.715037 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s677c\"" Sep 10 05:19:42.716065 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.716047 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 05:19:42.720138 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.720074 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j4jxp\"" Sep 10 05:19:42.720839 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.720823 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 05:19:42.722429 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.722408 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 05:19:42.727173 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.727149 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-t6jgh\"" Sep 10 05:19:42.742748 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.742717 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-q9fws_2f193ed6-cb83-4049-8589-e5e8513606fe/monitoring-plugin/0.log" Sep 10 05:19:42.744250 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.744230 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 05:19:42.750424 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.750386 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 05:19:42.756405 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.756378 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v49sd\"" Sep 10 05:19:42.771847 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.771813 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 05:19:42.777690 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.777662 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 05:19:42.777840 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.777707 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 05:19:42.779160 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.779144 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 05:19:42.783495 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.783441 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 05:19:42.788418 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.788388 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2lbl5_07f9db3c-da60-4f88-8327-61bd6936ecc3/node-exporter/0.log" Sep 10 05:19:42.788847 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.788830 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hwcjp\"" Sep 10 05:19:42.793397 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.793368 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 05:19:42.793569 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.793477 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 05:19:42.796314 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.796286 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 05:19:42.799635 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.799616 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9k46b\"" Sep 10 05:19:42.803103 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.803084 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vhrd6\"" Sep 10 05:19:42.807024 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.807005 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2lbl5_07f9db3c-da60-4f88-8327-61bd6936ecc3/kube-rbac-proxy/0.log" Sep 10 05:19:42.809759 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.809736 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 05:19:42.815272 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.815249 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 05:19:42.823802 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.823772 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kghkv\"" Sep 10 05:19:42.827826 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.827803 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2lbl5_07f9db3c-da60-4f88-8327-61bd6936ecc3/init-textfile/0.log" Sep 10 05:19:42.836056 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.836033 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 05:19:42.836335 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.836316 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 05:19:42.843211 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.843187 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 05:19:42.844731 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.844715 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 05:19:42.857646 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.857612 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9hw97\"" Sep 10 05:19:42.860222 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.860199 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 05:19:42.865327 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.865299 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 05:19:42.867889 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.867860 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 05:19:42.872497 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.872475 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 05:19:42.876770 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.876746 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 05:19:42.876941 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.876919 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fi2tqmklkgpsj\"" Sep 10 05:19:42.884603 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.884575 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 05:19:42.893792 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.893757 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksvhk\"" Sep 10 05:19:42.897909 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.897886 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-b6stt\"" Sep 10 05:19:42.909461 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.909418 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 05:19:42.921419 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.921391 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 05:19:42.928016 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.927992 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 05:19:42.928934 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.928918 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 05:19:42.939021 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.938993 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7k27c\"" Sep 10 05:19:42.939370 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.939354 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 05:19:42.940456 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.940432 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 05:19:42.949054 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.949020 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 05:19:42.950878 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.950856 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kf5vp\"" Sep 10 05:19:42.950995 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.950914 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 05:19:42.951695 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.951679 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 05:19:42.953349 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.953328 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zzfzt\"" Sep 10 05:19:42.954790 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.954772 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 05:19:42.957556 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.957539 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 05:19:42.959275 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.959258 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-z9tjc\"" Sep 10 05:19:42.976134 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.976056 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 05:19:42.977715 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.977696 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 05:19:42.981916 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.981896 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 05:19:42.986594 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.986572 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d4254\"" Sep 10 05:19:42.993229 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.993201 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 05:19:42.995255 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.995229 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 05:19:42.998745 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:42.998726 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gx84n\"" Sep 10 05:19:43.000197 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.000179 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 05:19:43.001517 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.001494 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 05:19:43.005389 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.005365 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 05:19:43.010439 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.010414 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pgx4x_d60fe704-87f9-4b56-8196-4354e0199fb3/kube-rbac-proxy-main/0.log" Sep 10 05:19:43.014380 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.014358 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 05:19:43.025800 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.025768 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 05:19:43.029319 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.029290 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cqf2m\"" Sep 10 05:19:43.031591 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.031556 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 05:19:43.033014 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.032979 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pgx4x_d60fe704-87f9-4b56-8196-4354e0199fb3/kube-rbac-proxy-self/0.log" Sep 10 05:19:43.033248 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.033233 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 05:19:43.036143 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.036122 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 05:19:43.042976 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.042948 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-756kd\"" Sep 10 05:19:43.044293 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.044273 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dzbdb\"" Sep 10 05:19:43.048882 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.048859 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 05:19:43.052321 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.052295 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-khd29\"" Sep 10 05:19:43.053390 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.053369 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 05:19:43.053527 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.053502 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 05:19:43.054578 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.054548 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pgx4x_d60fe704-87f9-4b56-8196-4354e0199fb3/openshift-state-metrics/0.log" Sep 10 05:19:43.054699 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.054682 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 05:19:43.062595 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.062563 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 05:19:43.088137 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.088104 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 05:19:43.088676 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.088651 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_517297a6-ebd6-4d0b-976e-707327e28eaa/prometheus/0.log" Sep 10 05:19:43.108403 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.108375 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_517297a6-ebd6-4d0b-976e-707327e28eaa/config-reloader/0.log" Sep 10 05:19:43.109211 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.109191 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 05:19:43.109290 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.109275 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 05:19:43.109714 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.109699 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 10 05:19:43.110012 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.109997 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 05:19:43.124796 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.124765 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 05:19:43.129871 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.129821 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_517297a6-ebd6-4d0b-976e-707327e28eaa/thanos-sidecar/0.log" Sep 10 05:19:43.134351 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.134326 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 05:19:43.144333 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.144305 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9hz5t\"" Sep 10 05:19:43.150932 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.150907 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 05:19:43.153709 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.153678 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_517297a6-ebd6-4d0b-976e-707327e28eaa/kube-rbac-proxy-web/0.log" Sep 10 05:19:43.158660 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.158635 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-25t8d\"" Sep 10 05:19:43.166537 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.166507 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 05:19:43.174760 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.174722 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_517297a6-ebd6-4d0b-976e-707327e28eaa/kube-rbac-proxy/0.log" Sep 10 05:19:43.189627 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.189596 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 10 05:19:43.189822 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.189680 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4f2wj\"" Sep 10 05:19:43.194332 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.194305 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-b9bc6\"" Sep 10 05:19:43.201661 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.201634 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_517297a6-ebd6-4d0b-976e-707327e28eaa/kube-rbac-proxy-thanos/0.log" Sep 10 05:19:43.202586 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.202568 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 10 05:19:43.207058 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.207033 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 05:19:43.215332 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.215300 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 05:19:43.219556 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.219529 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 05:19:43.223029 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.223007 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 05:19:43.225372 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.225352 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_517297a6-ebd6-4d0b-976e-707327e28eaa/init-config-reloader/0.log" Sep 10 05:19:43.226266 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.226212 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 05:19:43.229165 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.229145 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 05:19:43.232045 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.232021 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 05:19:43.232328 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.232312 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 05:19:43.238873 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.238844 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 05:19:43.243938 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.243911 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 05:19:43.244118 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.243944 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 05:19:43.244118 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.243963 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 05:19:43.257300 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.257272 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-k4vcd_d2137982-cc72-482c-bcdd-1520a94294e4/prometheus-operator/0.log" Sep 10 05:19:43.258208 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.258189 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 05:19:43.262319 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.262297 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 05:19:43.271229 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.271198 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 05:19:43.277543 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.277508 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 05:19:43.277543 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.277536 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7c7zc\"" Sep 10 05:19:43.279006 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.278979 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-k4vcd_d2137982-cc72-482c-bcdd-1520a94294e4/kube-rbac-proxy/0.log" Sep 10 05:19:43.284074 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.284046 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 05:19:43.285374 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.285351 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 05:19:43.291087 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.291059 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 05:19:43.291270 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.291126 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 05:19:43.293747 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.293722 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 05:19:43.298529 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.298503 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdnb2\"" Sep 10 05:19:43.299405 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.299387 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 05:19:43.305783 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.305758 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-kvpdd_97a4470f-9ff5-446d-81b7-79288f79b652/prometheus-operator-admission-webhook/0.log" Sep 10 05:19:43.308695 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.308674 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 05:19:43.308855 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.308837 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 05:19:43.317558 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.317531 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 05:19:43.322717 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.322694 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 05:19:43.326736 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.326707 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 05:19:43.327222 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.327205 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 05:19:43.332492 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.332465 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 05:19:43.337290 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.337260 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 05:19:43.337466 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.337338 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 05:19:43.351300 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.351263 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 05:19:43.352125 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.352100 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 05:19:43.354061 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.354036 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq"] Sep 10 05:19:43.354439 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.354425 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9557d6f-fdc3-4260-8863-cb628ebafa62" containerName="maas-api" Sep 10 05:19:43.354504 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.354440 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9557d6f-fdc3-4260-8863-cb628ebafa62" containerName="maas-api" Sep 10 05:19:43.354558 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.354548 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9557d6f-fdc3-4260-8863-cb628ebafa62" containerName="maas-api" Sep 10 05:19:43.357830 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.357807 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.361349 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.361326 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x7vl4\"/\"kube-root-ca.crt\"" Sep 10 05:19:43.362676 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.362658 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x7vl4\"/\"default-dockercfg-b6tnh\"" Sep 10 05:19:43.369504 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.369478 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x7vl4\"/\"openshift-service-ca.crt\"" Sep 10 05:19:43.409544 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.409515 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 05:19:43.429713 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.429684 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 05:19:43.449390 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.449360 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 05:19:43.470397 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.470366 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bifgalq23t6sr\"" Sep 10 05:19:43.490344 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.490314 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 05:19:43.510779 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.510745 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6nh906m7lfq4a\"" Sep 10 05:19:43.529903 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.529872 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 10 05:19:43.536896 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.536872 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d58df5db-tzss8_7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a/thanos-query/0.log" Sep 10 05:19:43.544325 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.544293 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjst\" (UniqueName: \"kubernetes.io/projected/87d18084-4e03-408f-98a9-f8b16e67a99e-kube-api-access-5pjst\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.544467 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.544337 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-proc\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.544467 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.544413 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-lib-modules\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.544572 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.544480 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-sys\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.544572 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.544534 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-podres\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.549439 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.549416 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 05:19:43.558686 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.558651 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d58df5db-tzss8_7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a/kube-rbac-proxy-web/0.log" Sep 10 05:19:43.569814 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.569784 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 05:19:43.582974 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.582948 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d58df5db-tzss8_7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a/kube-rbac-proxy/0.log" Sep 10 05:19:43.589910 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.589886 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m66hn\"" Sep 10 05:19:43.605510 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.605482 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d58df5db-tzss8_7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a/prom-label-proxy/0.log" Sep 10 05:19:43.619633 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.619600 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 05:19:43.629164 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.629129 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d58df5db-tzss8_7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a/kube-rbac-proxy-rules/0.log" Sep 10 05:19:43.629352 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.629204 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 05:19:43.645371 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.645336 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjst\" (UniqueName: \"kubernetes.io/projected/87d18084-4e03-408f-98a9-f8b16e67a99e-kube-api-access-5pjst\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.645371 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.645375 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-proc\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.645652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.645460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-lib-modules\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.645652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.645485 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-sys\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.645652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.645521 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-podres\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.645652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.645545 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-proc\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.645652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.645588 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-sys\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.645652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.645615 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-lib-modules\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.645652 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.645652 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/87d18084-4e03-408f-98a9-f8b16e67a99e-podres\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:43.649158 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.649134 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 05:19:43.650944 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.650917 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55d58df5db-tzss8_7dbcbb0f-9166-4ef6-9d40-a6eb74c35a5a/kube-rbac-proxy-metrics/0.log" Sep 10 05:19:43.669561 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.669528 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jbhx5\"" Sep 10 05:19:43.688995 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.688957 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 05:19:43.709129 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.709091 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 05:19:43.730151 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.730111 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 10 05:19:43.749607 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.749511 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bstcg\"" Sep 10 05:19:43.769976 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.769942 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 05:19:43.789668 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.789637 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dhnvv\"" Sep 10 05:19:43.809319 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.809292 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 05:19:43.830078 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.830049 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 05:19:43.848977 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.848931 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jksxr\"" Sep 10 05:19:43.869979 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.869948 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nj7q4\"" Sep 10 05:19:43.889064 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.889029 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 05:19:43.909731 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.909702 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vhk75\"" Sep 10 05:19:43.929695 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.929656 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 05:19:43.949167 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.949131 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 05:19:43.969527 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.969494 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jxftf\"" Sep 10 05:19:43.995712 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:43.995670 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjst\" (UniqueName: \"kubernetes.io/projected/87d18084-4e03-408f-98a9-f8b16e67a99e-kube-api-access-5pjst\") pod \"perf-node-gather-daemonset-qchzq\" (UID: \"87d18084-4e03-408f-98a9-f8b16e67a99e\") " pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:44.268275 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:44.268240 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:44.549697 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:44.549672 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq"] Sep 10 05:19:44.552187 ip-10-0-136-213 kubenswrapper[2572]: W0910 05:19:44.552149 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod87d18084_4e03_408f_98a9_f8b16e67a99e.slice/crio-9a81dd464aefca94ef4c8c04f5af290001e6a798b08001576d1865cddc845b6c WatchSource:0}: Error finding container 9a81dd464aefca94ef4c8c04f5af290001e6a798b08001576d1865cddc845b6c: Status 404 returned error can't find the container with id 9a81dd464aefca94ef4c8c04f5af290001e6a798b08001576d1865cddc845b6c Sep 10 05:19:44.745433 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:44.745398 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" event={"ID":"87d18084-4e03-408f-98a9-f8b16e67a99e","Type":"ContainerStarted","Data":"7378d26f8690da5a7b4df2642141952ee1a6586eb4fc96761242e69c393c5b90"} Sep 10 05:19:44.745656 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:44.745489 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:44.745656 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:44.745515 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" event={"ID":"87d18084-4e03-408f-98a9-f8b16e67a99e","Type":"ContainerStarted","Data":"9a81dd464aefca94ef4c8c04f5af290001e6a798b08001576d1865cddc845b6c"} Sep 10 05:19:44.762059 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:44.761760 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" podStartSLOduration=36.76173581 podStartE2EDuration="36.76173581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 05:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 05:19:44.761271237 +0000 UTC m=+2304.265104299" watchObservedRunningTime="2026-09-10 05:19:44.76173581 +0000 UTC m=+2304.265568872" Sep 10 05:19:45.806883 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:45.806852 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-44wrh_01e9fb97-7654-4ee0-88f3-baeef295cfbe/download-server/0.log" Sep 10 05:19:47.594544 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:47.594510 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pcllp_7da0728c-18d9-4d7b-9122-0fccd5a6c298/dns/0.log" Sep 10 05:19:47.613814 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:47.613781 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pcllp_7da0728c-18d9-4d7b-9122-0fccd5a6c298/kube-rbac-proxy/0.log" Sep 10 05:19:47.767185 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:47.767153 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientMemory" Sep 10 05:19:47.767185 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:47.767189 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 05:19:47.767412 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:47.767203 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeHasSufficientPID" Sep 10 05:19:47.767412 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:47.767220 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-213.ec2.internal" event="NodeReady" Sep 10 05:19:47.819033 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:47.818996 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z2nhq_c2506566-130f-4ac1-9be0-b67224951c9d/dns-node-resolver/0.log" Sep 10 05:19:48.398710 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:48.398678 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-clkch_b773f2a8-5f9d-4a6f-850f-3c37ae6f4705/node-ca/0.log" Sep 10 05:19:49.352365 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:49.352336 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-5wzqp_c37350ae-7be6-421a-a2ce-0ee47bacd479/istio-proxy/0.log" Sep 10 05:19:49.824355 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:49.824317 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dgxnt_fae80c42-bbf1-4d41-a1d4-87d5dfd4ee1b/discovery/0.log" Sep 10 05:19:49.919363 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:49.919334 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9l5xw_716fa502-5c07-487c-9410-4546ddfffec2/istio-proxy/0.log" Sep 10 05:19:49.924504 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:49.924472 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9l5xw_716fa502-5c07-487c-9410-4546ddfffec2/istio-proxy/1.log" Sep 10 05:19:50.001870 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:50.001835 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4nxnk_cf246a19-caf3-4fd6-8611-add57e445231/payload-processing/0.log" Sep 10 05:19:50.760861 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:50.760836 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x7vl4/perf-node-gather-daemonset-qchzq" Sep 10 05:19:50.932421 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:50.932391 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8ffk8_c6812836-3390-4aa9-b1b3-b997d53e56fb/serve-healthcheck-canary/0.log" Sep 10 05:19:51.742176 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:51.741897 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sncsj_cc6b935a-1508-46df-9efe-ce6f1bd0da04/kube-rbac-proxy/0.log" Sep 10 05:19:51.766133 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:51.766088 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sncsj_cc6b935a-1508-46df-9efe-ce6f1bd0da04/exporter/0.log" Sep 10 05:19:51.800752 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:51.800721 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sncsj_cc6b935a-1508-46df-9efe-ce6f1bd0da04/extractor/0.log" Sep 10 05:19:54.647826 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:54.647787 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7597cb784c-nvv8r_a759187e-0dfc-4b81-8b64-a8c657aba792/manager/0.log" Sep 10 05:19:54.822969 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:54.822936 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c57f4866c-g8gh8_75eeed64-56c0-47e8-a25f-c61c067b69e3/manager/0.log" Sep 10 05:19:54.851605 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:54.851573 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c57f4866c-g8gh8_75eeed64-56c0-47e8-a25f-c61c067b69e3/copy-manifests/0.log" Sep 10 05:19:56.133549 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:56.133520 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5fb77b59f6-6lnfd_469671b2-66b7-4971-9bcc-6c79151f8072/manager/0.log" Sep 10 05:19:56.178466 ip-10-0-136-213 kubenswrapper[2572]: I0910 05:19:56.178344 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-tn44q_48b3454d-b1dd-4681-8b47-6e7222779d75/openshift-lws-operator/0.log"