Sep 16 01:13:59.936740 ip-10-0-142-56 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 01:13:59.936757 ip-10-0-142-56 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 01:13:59.936766 ip-10-0-142-56 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 01:13:59.936999 ip-10-0-142-56 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 01:14:10.102546 ip-10-0-142-56 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 01:14:10.102565 ip-10-0-142-56 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 62acf043e08f42c2aec2b6a87ec25f7c -- Sep 16 01:15:49.297649 ip-10-0-142-56 systemd[1]: Starting Kubernetes Kubelet... Sep 16 01:15:49.729734 ip-10-0-142-56 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 01:15:49.729734 ip-10-0-142-56 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 01:15:49.729734 ip-10-0-142-56 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 01:15:49.729734 ip-10-0-142-56 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 01:15:49.729734 ip-10-0-142-56 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 01:15:49.730376 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.729754 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 01:15:49.733756 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733738 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:15:49.733756 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733757 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733761 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733765 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733768 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733771 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733775 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733778 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733781 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733783 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733786 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733789 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733791 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733794 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733797 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733799 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733802 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733805 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733807 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733810 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733812 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:15:49.733831 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733815 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733817 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733820 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733822 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733825 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733827 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733830 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733832 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733835 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733837 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733840 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733844 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733847 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733849 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733852 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733854 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733857 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733859 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733862 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733864 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:15:49.734362 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733867 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733869 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733872 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733874 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733876 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733879 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733881 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733883 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733886 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733888 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733890 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733893 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733896 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733898 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733901 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733905 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733907 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733911 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733915 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:15:49.735121 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733918 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733920 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733923 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733925 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733928 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733930 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733933 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733935 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733938 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733940 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733942 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733946 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733952 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733955 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733958 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733961 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733964 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733966 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733969 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:15:49.735693 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733971 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733973 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733976 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733979 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733982 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733984 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733986 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733990 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733993 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733996 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.733998 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.734001 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.734003 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.734006 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.734008 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.734010 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.734013 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.734016 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.734018 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:15:49.736147 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.734020 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736393 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736405 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736411 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736414 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736417 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736421 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736426 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736429 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736432 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736435 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736438 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736440 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736443 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736445 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736448 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736451 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736453 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736456 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736458 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:15:49.736616 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736461 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736463 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736465 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736468 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736470 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736472 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736476 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736478 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736482 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736484 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736486 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736489 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736491 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736494 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736497 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736501 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736504 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736507 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736510 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:15:49.737124 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736512 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736514 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736517 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736520 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736522 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736524 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736526 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736529 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736531 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736534 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736536 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736538 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736541 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736544 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736546 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736549 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736551 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736553 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736556 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736558 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:15:49.737634 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736561 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736564 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736566 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736568 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736572 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736574 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736577 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736580 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736582 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736585 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736588 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736590 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736593 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736596 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736598 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736601 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736605 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736609 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736612 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:15:49.738123 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736614 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736617 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736620 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736622 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736625 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736628 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736631 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736634 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736636 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736639 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736641 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736644 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736646 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736649 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736651 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736654 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736657 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736660 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736662 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:15:49.738591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736665 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736667 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.736670 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737053 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737057 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737061 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737064 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737066 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737069 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737071 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737074 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737077 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737080 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737082 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737085 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737087 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737090 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737092 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737094 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737098 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:15:49.739052 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737101 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737103 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737106 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737108 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737111 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737113 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737117 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737121 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737125 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737128 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737131 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737134 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737137 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737140 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737143 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737145 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737148 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737150 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737153 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:15:49.739559 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737155 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737158 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737160 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737163 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737165 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737168 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737170 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737173 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737176 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737179 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737182 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737186 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737189 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737192 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737195 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737198 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737200 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737203 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737205 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:15:49.740027 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737208 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737210 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737213 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737215 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737218 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737220 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737224 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737227 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737230 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737232 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737235 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737237 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737240 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737262 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737265 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737268 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737271 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737295 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737298 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:15:49.740542 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737301 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737304 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737307 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737310 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737313 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737316 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737318 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737321 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737325 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737327 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737330 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737332 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737335 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737337 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737340 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737343 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737346 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737348 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737350 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737353 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:15:49.741114 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737355 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737357 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737360 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737362 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.737365 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737439 2567 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737480 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737488 2567 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737492 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737497 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737500 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737504 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737509 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737513 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737516 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737520 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737523 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737526 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737529 2567 flags.go:64] FLAG: --cgroup-root="" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737531 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737535 2567 flags.go:64] FLAG: --client-ca-file="" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737538 2567 flags.go:64] FLAG: --cloud-provider="external" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737541 2567 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737550 2567 flags.go:64] FLAG: --cluster-domain="" Sep 16 01:15:49.741627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737553 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737556 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737562 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737566 2567 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737573 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737576 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737579 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737582 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737585 2567 flags.go:64] FLAG: --contention-profiling="false" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737588 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737591 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737594 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737597 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737601 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737603 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737606 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737610 2567 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737613 2567 flags.go:64] FLAG: --enable-server="true" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737615 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737619 2567 flags.go:64] FLAG: --event-burst="100" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737622 2567 flags.go:64] FLAG: --event-qps="50" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737625 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737630 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737632 2567 flags.go:64] FLAG: --eviction-hard="" Sep 16 01:15:49.742206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737636 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737639 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737642 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737645 2567 flags.go:64] FLAG: --eviction-soft="" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737648 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737651 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737655 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737658 2567 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737660 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737663 2567 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737667 2567 flags.go:64] FLAG: --feature-gates="" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737671 2567 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737673 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737676 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737679 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737682 2567 flags.go:64] FLAG: --healthz-port="10248" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737686 2567 flags.go:64] FLAG: --help="false" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737689 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-142-56.ec2.internal" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737692 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737694 2567 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737697 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737701 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737704 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737706 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 01:15:49.742780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737709 2567 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737712 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737715 2567 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737717 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737721 2567 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737724 2567 flags.go:64] FLAG: --kube-reserved="" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737728 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737731 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737734 2567 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737736 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737739 2567 flags.go:64] FLAG: --lock-file="" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737742 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737744 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737747 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737753 2567 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737756 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737759 2567 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737762 2567 flags.go:64] FLAG: --logging-format="text" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737766 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737769 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737772 2567 flags.go:64] FLAG: --manifest-url="" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737775 2567 flags.go:64] FLAG: --manifest-url-header="" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737779 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737783 2567 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737787 2567 flags.go:64] FLAG: --max-pods="110" Sep 16 01:15:49.743412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737790 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737793 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737796 2567 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737799 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737802 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737804 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737807 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737815 2567 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737818 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737821 2567 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737824 2567 flags.go:64] FLAG: --pod-cidr="" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737827 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737833 2567 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737836 2567 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737840 2567 flags.go:64] FLAG: --pods-per-core="0" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737843 2567 flags.go:64] FLAG: --port="10250" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737846 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737849 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bd638c337b1a295b" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737852 2567 flags.go:64] FLAG: --qos-reserved="" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737855 2567 flags.go:64] FLAG: --read-only-port="10255" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737858 2567 flags.go:64] FLAG: --register-node="true" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737861 2567 flags.go:64] FLAG: --register-with-taints="" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737864 2567 flags.go:64] FLAG: --registry-burst="10" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737867 2567 flags.go:64] FLAG: --registry-qps="5" Sep 16 01:15:49.744058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737871 2567 flags.go:64] FLAG: --reserved-cpus="" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737875 2567 flags.go:64] FLAG: --reserved-memory="" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737878 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737881 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737885 2567 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737887 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737890 2567 flags.go:64] FLAG: --runonce="false" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737893 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737896 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737900 2567 flags.go:64] FLAG: --seccomp-default="false" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737902 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737905 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737908 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737911 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737914 2567 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737917 2567 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737920 2567 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737923 2567 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737925 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737929 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737932 2567 flags.go:64] FLAG: --system-cgroups="" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737935 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737942 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737945 2567 flags.go:64] FLAG: --tls-cert-file="" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737948 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 01:15:49.744658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737951 2567 flags.go:64] FLAG: --tls-min-version="" Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737953 2567 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737956 2567 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737959 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737962 2567 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737965 2567 flags.go:64] FLAG: --v="2" Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737969 2567 flags.go:64] FLAG: --version="false" Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737973 2567 flags.go:64] FLAG: --vmodule="" Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737979 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.737983 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738100 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738104 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738108 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738111 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738115 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738118 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738121 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738123 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738126 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738129 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738132 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738135 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:15:49.745332 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738137 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738140 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738142 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738145 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738147 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738150 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738152 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738157 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738159 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738162 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738164 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738167 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738170 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738173 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738175 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738178 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738180 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738183 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738188 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:15:49.746251 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738191 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738193 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738196 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738199 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738201 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738204 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738206 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738209 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738212 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738214 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738217 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738220 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738222 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738225 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738227 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738230 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738233 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738235 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738238 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738240 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:15:49.747086 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738245 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738248 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738250 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738252 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738255 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738259 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738262 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738265 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738267 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738270 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738291 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738298 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738302 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738305 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738308 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738311 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738314 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738317 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738320 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:15:49.747979 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738322 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738325 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738328 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738331 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738334 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738338 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738340 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738343 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738346 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738349 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738352 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738355 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738357 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738361 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738364 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738367 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738369 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738372 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738374 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738377 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:15:49.748825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738379 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738381 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738384 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738387 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738390 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738393 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738396 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738399 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.738410 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.739209 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.747673 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.747695 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.747721 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.747729 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747841 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:15:49.749570 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747849 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747854 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747860 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747864 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747870 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747875 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747880 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747885 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747890 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747894 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747898 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747903 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747907 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747911 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747916 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747920 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747925 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747929 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747933 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:15:49.750024 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747937 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747942 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747946 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747950 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747954 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747959 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747963 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747967 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747971 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747977 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747981 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747985 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747989 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747994 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.747998 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748002 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748006 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748011 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748015 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748020 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:15:49.750825 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748024 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748028 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748032 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748036 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748040 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748047 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748053 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748060 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748065 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748071 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748075 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748080 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748085 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748090 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748094 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748101 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748108 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748114 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748119 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:15:49.751666 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748123 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748127 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748133 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748139 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748143 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748147 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748151 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748155 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748160 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748164 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748168 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748172 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748177 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748180 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748185 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748189 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748193 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748198 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748202 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748206 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:15:49.752165 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748210 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748214 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748218 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748223 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748227 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748231 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748235 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748239 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748243 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748247 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748251 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748255 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748259 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748264 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748269 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748293 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748299 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748304 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748308 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:15:49.752788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748312 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.748320 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748504 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748513 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748518 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748523 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748528 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748533 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748538 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748542 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748546 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748550 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748554 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748559 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748563 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 01:15:49.753260 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748567 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748571 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748575 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748579 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748583 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748587 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748591 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748595 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748599 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748603 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748607 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748612 2567 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748616 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748621 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748625 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748630 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748635 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748639 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748643 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748648 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748652 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 01:15:49.753713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748656 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748660 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748664 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748668 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748672 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748676 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748680 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748684 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748689 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748693 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748698 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748702 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748708 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748714 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748718 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748723 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748727 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748734 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 01:15:49.754205 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748739 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748744 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748749 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748753 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748758 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748763 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748768 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748772 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748777 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748781 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748786 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748790 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748794 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748800 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748806 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748811 2567 feature_gate.go:328] unrecognized feature gate: Example Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748815 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748819 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748823 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 01:15:49.754713 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748826 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748830 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748834 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748838 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748841 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748845 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748850 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748854 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748858 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748862 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748866 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748871 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748876 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748880 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748884 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748890 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748894 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748898 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748903 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748907 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 01:15:49.755167 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748911 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748915 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748919 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748926 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748930 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748934 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748939 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:49.748943 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.748950 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.749132 2567 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.752159 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.753418 2567 server.go:1019] "Starting client certificate rotation" Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.753528 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 01:15:49.755684 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.753568 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 01:15:49.756584 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.756567 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 01:15:49.757757 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.757735 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 01:15:49.770191 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.770176 2567 log.go:25] "Validated CRI v1 runtime API" Sep 16 01:15:49.773251 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.773234 2567 log.go:25] "Validated CRI v1 image API" Sep 16 01:15:49.774093 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.774072 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 01:15:49.780470 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.780452 2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e6f1525c-1a5e-4e64-b6b3-ada78ec6c6a2:/dev/nvme0n1p3 f71e5cd8-8f54-4009-b6d5-6a18307805b4:/dev/nvme0n1p4] Sep 16 01:15:49.780527 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.780470 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 01:15:49.784069 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.784047 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 01:15:49.785915 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.785806 2567 manager.go:216] Machine: {Timestamp:2026-09-16 01:15:49.784077755 +0000 UTC m=+0.369879959 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097708 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b95b38b482426e9b7279ec787d5a4 SystemUUID:ec2b95b3-8b48-2426-e9b7-279ec787d5a4 BootID:62acf043-e08f-42c2-aec2-b6a87ec25f7c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cf:3f:13:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cf:3f:13:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:9e:fe:0b:65:d4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 01:15:49.785915 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.785913 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 01:15:49.786031 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.786023 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 01:15:49.789118 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.789091 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 01:15:49.789250 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.789120 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-56.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 01:15:49.790452 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.790442 2567 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 01:15:49.790492 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.790455 2567 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 01:15:49.790492 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.790462 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 01:15:49.791116 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.791106 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 01:15:49.792061 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.792052 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 01:15:49.796611 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.796599 2567 state_mem.go:36] "Initialized new in-memory state store" Sep 16 01:15:49.796716 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.796706 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 01:15:49.796748 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.796735 2567 kubelet.go:486] "Attempting to sync node with API server" Sep 16 01:15:49.796748 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.796745 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 01:15:49.796797 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.796759 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 01:15:49.796797 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.796768 2567 kubelet.go:398] "Adding apiserver pod source" Sep 16 01:15:49.796797 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.796776 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 01:15:49.797113 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.797101 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 01:15:49.797180 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.797123 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 01:15:49.799413 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.799397 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 01:15:49.802340 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.802327 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 01:15:49.802401 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.802357 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 01:15:49.804323 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804309 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 01:15:49.804372 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804328 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 01:15:49.804372 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804335 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 01:15:49.804372 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804340 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 01:15:49.804372 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804349 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 01:15:49.804372 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804354 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 01:15:49.804372 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804372 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 01:15:49.804523 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804377 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 01:15:49.804523 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804383 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 01:15:49.804523 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804388 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 01:15:49.804523 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804400 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 01:15:49.804523 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.804405 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 01:15:49.805220 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.805210 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 01:15:49.805220 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.805220 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 01:15:49.808730 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.808681 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 01:15:49.808915 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.808893 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-56.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 01:15:49.809901 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.809882 2567 server.go:1268] "Started kubelet" Sep 16 01:15:49.810162 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.810123 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 01:15:49.810771 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.810708 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 01:15:49.810853 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.810824 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 01:15:49.811245 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.811226 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-56.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 01:15:49.811852 ip-10-0-142-56 systemd[1]: Started Kubernetes Kubelet. Sep 16 01:15:49.813082 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.813065 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 01:15:49.813256 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.813245 2567 server.go:310] "Adding debug handlers to kubelet server" Sep 16 01:15:49.816241 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.816222 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 01:15:49.819968 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.818880 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-56.ec2.internal.18d5a82fb5630ab6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-56.ec2.internal,UID:ip-10-0-142-56.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-56.ec2.internal,},FirstTimestamp:2026-09-16 01:15:49.809736374 +0000 UTC m=+0.395538578,LastTimestamp:2026-09-16 01:15:49.809736374 +0000 UTC m=+0.395538578,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-56.ec2.internal,}" Sep 16 01:15:49.821174 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.821110 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 01:15:49.822110 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.822092 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 01:15:49.822196 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.822180 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-56.ec2.internal\" not found" Sep 16 01:15:49.822829 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.822797 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 01:15:49.822829 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.822817 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 01:15:49.822967 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.822795 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 01:15:49.822967 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.822913 2567 reconstruct.go:97] "Volume reconstruction finished" Sep 16 01:15:49.822967 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.822923 2567 reconciler.go:29] "Reconciler: start to sync state" Sep 16 01:15:49.822967 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.822923 2567 factory.go:55] Registering systemd factory Sep 16 01:15:49.822967 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.822947 2567 factory.go:223] Registration of the systemd container factory successfully Sep 16 01:15:49.823183 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.823060 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s8gw6" Sep 16 01:15:49.823183 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.823150 2567 factory.go:153] Registering CRI-O factory Sep 16 01:15:49.823183 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.823159 2567 factory.go:223] Registration of the crio container factory successfully Sep 16 01:15:49.823351 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.823204 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 01:15:49.823351 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.823217 2567 factory.go:103] Registering Raw factory Sep 16 01:15:49.823351 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.823228 2567 manager.go:1196] Started watching for new ooms in manager Sep 16 01:15:49.823612 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.823590 2567 manager.go:349] Starting recovery of all containers Sep 16 01:15:49.831446 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.831375 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s8gw6" Sep 16 01:15:49.832640 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.832622 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 01:15:49.832715 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.832700 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-56.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 01:15:49.834100 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.834082 2567 manager.go:354] Recovery completed Sep 16 01:15:49.838114 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.838099 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:15:49.839889 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.839869 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:15:49.839963 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.839911 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:15:49.839963 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.839924 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:15:49.840677 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.840664 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 01:15:49.840677 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.840676 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 01:15:49.840797 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.840681 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:15:49.840797 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.840689 2567 state_mem.go:36] "Initialized new in-memory state store" Sep 16 01:15:49.841491 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.841470 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:15:49.841580 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.841504 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:15:49.841580 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.841516 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:15:49.842673 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.842661 2567 policy_none.go:49] "None policy: Start" Sep 16 01:15:49.842740 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.842679 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 01:15:49.842740 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.842692 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 01:15:49.844613 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.844600 2567 policy_none.go:47] "Start" Sep 16 01:15:49.876943 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.876813 2567 manager.go:337] "Starting Device Plugin manager" Sep 16 01:15:49.877058 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.876974 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 01:15:49.877058 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.876987 2567 server.go:87] "Starting device plugin registration server" Sep 16 01:15:49.877332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.877319 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 01:15:49.877413 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.877335 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 01:15:49.877478 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.877409 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 01:15:49.877529 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.877499 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 01:15:49.877529 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.877509 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 01:15:49.878066 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.878046 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 01:15:49.878150 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.878097 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-56.ec2.internal\" not found" Sep 16 01:15:49.931303 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.931259 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 01:15:49.933374 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.932591 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 01:15:49.933374 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.932638 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 01:15:49.933374 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.932660 2567 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 01:15:49.933374 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.932735 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 01:15:49.934686 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.934667 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:15:49.977698 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.977669 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:15:49.978682 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.978664 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:15:49.978774 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.978701 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:15:49.978774 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.978713 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:15:49.978774 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.978736 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-56.ec2.internal" Sep 16 01:15:49.991155 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:49.991088 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-56.ec2.internal" Sep 16 01:15:49.991155 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:49.991112 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-56.ec2.internal\": node \"ip-10-0-142-56.ec2.internal\" not found" Sep 16 01:15:50.013506 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.013473 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-56.ec2.internal\" not found" Sep 16 01:15:50.032977 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.032935 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-56.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal"] Sep 16 01:15:50.033036 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.033019 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:15:50.034601 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.034586 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:15:50.034694 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.034616 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:15:50.034694 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.034626 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:15:50.036656 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.036644 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:15:50.036794 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.036765 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.036892 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.036801 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:15:50.037405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.037389 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:15:50.037501 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.037423 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:15:50.037501 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.037445 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:15:50.037501 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.037391 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:15:50.037501 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.037485 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:15:50.037501 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.037495 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:15:50.039744 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.039730 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.039810 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.039759 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 01:15:50.040797 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.040767 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientMemory" Sep 16 01:15:50.040888 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.040801 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 01:15:50.040888 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.040815 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeHasSufficientPID" Sep 16 01:15:50.079294 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.079252 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-56.ec2.internal\" not found" node="ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.083824 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.083806 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-56.ec2.internal\" not found" node="ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.114615 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.114584 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-56.ec2.internal\" not found" Sep 16 01:15:50.125615 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.125589 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ee9021a69ef53dd07b6a5e501d5f066-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal\" (UID: \"4ee9021a69ef53dd07b6a5e501d5f066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.125686 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.125618 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af880a3cf392188e514f562daba8d34c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-56.ec2.internal\" (UID: \"af880a3cf392188e514f562daba8d34c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.125686 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.125635 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ee9021a69ef53dd07b6a5e501d5f066-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal\" (UID: \"4ee9021a69ef53dd07b6a5e501d5f066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.214727 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.214689 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-56.ec2.internal\" not found" Sep 16 01:15:50.226105 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.226079 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af880a3cf392188e514f562daba8d34c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-56.ec2.internal\" (UID: \"af880a3cf392188e514f562daba8d34c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.226171 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.226114 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ee9021a69ef53dd07b6a5e501d5f066-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal\" (UID: \"4ee9021a69ef53dd07b6a5e501d5f066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.226171 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.226138 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ee9021a69ef53dd07b6a5e501d5f066-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal\" (UID: \"4ee9021a69ef53dd07b6a5e501d5f066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.226233 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.226190 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af880a3cf392188e514f562daba8d34c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-56.ec2.internal\" (UID: \"af880a3cf392188e514f562daba8d34c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.226233 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.226214 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4ee9021a69ef53dd07b6a5e501d5f066-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal\" (UID: \"4ee9021a69ef53dd07b6a5e501d5f066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.226316 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.226226 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ee9021a69ef53dd07b6a5e501d5f066-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal\" (UID: \"4ee9021a69ef53dd07b6a5e501d5f066\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.315537 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.315461 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-56.ec2.internal\" not found" Sep 16 01:15:50.382061 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.382028 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.386893 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.386767 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.416256 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.416223 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-56.ec2.internal\" not found" Sep 16 01:15:50.516887 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.516833 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-56.ec2.internal\" not found" Sep 16 01:15:50.617413 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.617330 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-56.ec2.internal\" not found" Sep 16 01:15:50.624964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.624943 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:15:50.722481 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.722455 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.733017 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.732999 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 01:15:50.734738 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.734721 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" Sep 16 01:15:50.744537 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.744515 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 01:15:50.752674 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.752652 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 01:15:50.752817 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.752796 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 01:15:50.797192 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.797141 2567 apiserver.go:52] "Watching apiserver" Sep 16 01:15:50.804180 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.804155 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 01:15:50.806093 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.806069 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-qcv88","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55","openshift-image-registry/node-ca-sxdzj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal","openshift-multus/multus-fxrml","openshift-network-operator/iptables-alerter-h92p2","kube-system/global-pull-secret-syncer-btbxj","kube-system/kube-apiserver-proxy-ip-10-0-142-56.ec2.internal","openshift-cluster-node-tuning-operator/tuned-s6rxq","openshift-dns/node-resolver-9glj9","openshift-multus/multus-additional-cni-plugins-78x9p","openshift-multus/network-metrics-daemon-snt2b","openshift-network-diagnostics/network-check-target-xgkwn","openshift-ovn-kubernetes/ovnkube-node-79tjp"] Sep 16 01:15:50.808779 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.808755 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:15:50.810905 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.810881 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.811202 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.811180 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 01:15:50.811291 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.811253 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xnqcw\"" Sep 16 01:15:50.811372 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.811339 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 01:15:50.813244 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.813219 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 01:15:50.813368 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.813294 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sxdzj" Sep 16 01:15:50.813368 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.813325 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-chz9k\"" Sep 16 01:15:50.813497 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.813222 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 01:15:50.813793 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.813776 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 01:15:50.815607 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.815586 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 01:15:50.815691 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.815661 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.815985 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.815966 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 01:15:50.816044 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.816016 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 01:15:50.816290 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.816262 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bxwm5\"" Sep 16 01:15:50.817910 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.817889 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h92p2" Sep 16 01:15:50.820359 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.820297 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 01:15:50.820468 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.820393 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 01:15:50.820468 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.820409 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 01:15:50.820648 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.820630 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:50.820738 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.820723 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ttcvx\"" Sep 16 01:15:50.820812 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.820714 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:15:50.821235 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.821212 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 01:15:50.821375 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.821339 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 01:15:50.821446 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.821342 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 01:15:50.821887 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.821871 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 01:15:50.822578 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.822563 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 01:15:50.822764 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.822750 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jmhnr\"" Sep 16 01:15:50.823347 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.823331 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.825615 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.825602 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 01:15:50.825798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.825782 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 01:15:50.825985 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.825967 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9glj9" Sep 16 01:15:50.826540 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.826031 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f7cx7\"" Sep 16 01:15:50.828168 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828150 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-98ht7\"" Sep 16 01:15:50.828441 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828422 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 01:15:50.828510 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828492 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 01:15:50.828731 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828670 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/711caeef-4722-4a09-b5df-55262773ebb3-cni-binary-copy\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.828731 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828696 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-hostroot\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.828731 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828715 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-etc-selinux\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.828843 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828733 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkw8\" (UniqueName: \"kubernetes.io/projected/7ff6c377-123d-427c-9f57-748013cb3bcd-kube-api-access-zgkw8\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.828843 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828772 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-multus-cni-dir\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.828843 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828801 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jszh\" (UniqueName: \"kubernetes.io/projected/cb26b3d5-1523-4e94-b72c-8a931b24e8a2-kube-api-access-9jszh\") pod \"iptables-alerter-h92p2\" (UID: \"cb26b3d5-1523-4e94-b72c-8a931b24e8a2\") " pod="openshift-network-operator/iptables-alerter-h92p2" Sep 16 01:15:50.828843 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828816 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-kubernetes\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.828843 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828835 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-lib-modules\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.828980 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828853 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-socket-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.828980 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828867 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-var-lib-kubelet\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.828980 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828881 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-sysctl-conf\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.828980 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828900 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-var-lib-kubelet\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.828980 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828922 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-host\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.828980 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828937 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prmzl\" (UniqueName: \"kubernetes.io/projected/711caeef-4722-4a09-b5df-55262773ebb3-kube-api-access-prmzl\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.828980 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828952 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-run\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.828986 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d7ce9c33-5241-41d5-8b0f-129237867308-etc-tuned\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829012 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm8gk\" (UniqueName: \"kubernetes.io/projected/d7ce9c33-5241-41d5-8b0f-129237867308-kube-api-access-qm8gk\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829049 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc623d5a-e0b2-4aa5-967f-c8b417121518-serviceca\") pod \"node-ca-sxdzj\" (UID: \"dc623d5a-e0b2-4aa5-967f-c8b417121518\") " pod="openshift-image-registry/node-ca-sxdzj" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829077 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-multus-socket-dir-parent\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829106 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-run-k8s-cni-cncf-io\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829128 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5lp5\" (UniqueName: \"kubernetes.io/projected/dc623d5a-e0b2-4aa5-967f-c8b417121518-kube-api-access-h5lp5\") pod \"node-ca-sxdzj\" (UID: \"dc623d5a-e0b2-4aa5-967f-c8b417121518\") " pod="openshift-image-registry/node-ca-sxdzj" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829148 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-var-lib-cni-bin\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829163 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-multus-conf-dir\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829178 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-etc-kubernetes\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829206 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb26b3d5-1523-4e94-b72c-8a931b24e8a2-iptables-alerter-script\") pod \"iptables-alerter-h92p2\" (UID: \"cb26b3d5-1523-4e94-b72c-8a931b24e8a2\") " pod="openshift-network-operator/iptables-alerter-h92p2" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829221 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb26b3d5-1523-4e94-b72c-8a931b24e8a2-host-slash\") pod \"iptables-alerter-h92p2\" (UID: \"cb26b3d5-1523-4e94-b72c-8a931b24e8a2\") " pod="openshift-network-operator/iptables-alerter-h92p2" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829240 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ce9c33-5241-41d5-8b0f-129237867308-tmp\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.829268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829265 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea88e142-7158-477d-b67a-1f1f450cbc6a-dbus\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829309 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c81a486-0ae1-4caa-be44-9a6cc0edf50c-agent-certs\") pod \"konnectivity-agent-qcv88\" (UID: \"5c81a486-0ae1-4caa-be44-9a6cc0edf50c\") " pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829335 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829357 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-run-netns\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829371 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-var-lib-cni-multus\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829387 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-run-multus-certs\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829400 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-sysconfig\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829421 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-systemd\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829459 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea88e142-7158-477d-b67a-1f1f450cbc6a-kubelet-config\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829485 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829507 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-registration-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829528 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-sys-fs\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829551 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/711caeef-4722-4a09-b5df-55262773ebb3-multus-daemon-config\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829573 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-modprobe-d\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829594 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-sysctl-d\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829617 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-sys\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.829717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829641 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c81a486-0ae1-4caa-be44-9a6cc0edf50c-konnectivity-ca\") pod \"konnectivity-agent-qcv88\" (UID: \"5c81a486-0ae1-4caa-be44-9a6cc0edf50c\") " pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:15:50.830220 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829663 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-device-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.830220 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829684 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc623d5a-e0b2-4aa5-967f-c8b417121518-host\") pod \"node-ca-sxdzj\" (UID: \"dc623d5a-e0b2-4aa5-967f-c8b417121518\") " pod="openshift-image-registry/node-ca-sxdzj" Sep 16 01:15:50.830220 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829706 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-system-cni-dir\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.830220 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829727 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-cnibin\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.830220 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.829807 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:50.831925 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.831903 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 01:15:50.832142 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.832120 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:50.832243 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.832184 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:15:50.835628 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.833642 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 01:15:50.835628 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.833735 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x6tx4\"" Sep 16 01:15:50.836158 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.836140 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 01:15:50.836591 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.836577 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:15:50.836661 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.836643 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:15:50.837243 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.837202 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 01:10:49 +0000 UTC" deadline="2028-04-13 02:53:07.319095 +0000 UTC" Sep 16 01:15:50.837317 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.837245 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13801h37m16.481853943s" Sep 16 01:15:50.838894 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.838879 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.841381 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.841357 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 01:15:50.841381 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.841370 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 01:15:50.841381 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.841376 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 01:15:50.841631 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.841414 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 01:15:50.841631 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.841445 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8s45n\"" Sep 16 01:15:50.841631 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.841545 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 01:15:50.841631 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.841558 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 01:15:50.841895 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.841883 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:15:50.854134 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.854114 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bhfsn" Sep 16 01:15:50.861370 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.861351 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bhfsn" Sep 16 01:15:50.891138 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.891117 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:15:50.925868 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.924879 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 01:15:50.930705 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930685 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:50.930809 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930722 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.930809 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930748 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-run-netns\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.930809 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930770 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-var-lib-cni-multus\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.930809 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-run-multus-certs\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930812 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-sysconfig\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930826 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930832 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-run-netns\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930835 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-run-openvswitch\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930871 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-var-lib-cni-multus\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930890 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a1f2830-2236-4dab-bfed-a19ac01c8948-ovn-node-metrics-cert\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930891 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-run-multus-certs\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930909 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-sysconfig\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930918 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930948 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-sys-fs\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930969 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/711caeef-4722-4a09-b5df-55262773ebb3-multus-daemon-config\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930981 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-sys-fs\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.930989 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-sysctl-d\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931010 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-sys\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.931011 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.931016 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931026 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vf9f\" (UniqueName: \"kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f\") pod \"network-check-target-xgkwn\" (UID: \"a8146604-b0d8-41a2-bd14-3daff2e45882\") " pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931041 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-cni-bin\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931062 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-cni-netd\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931103 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-device-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931116 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-sys\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931125 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-sysctl-d\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:50.931141 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret podName:ea88e142-7158-477d-b67a-1f1f450cbc6a nodeName:}" failed. No retries permitted until 2026-09-16 01:15:51.431118939 +0000 UTC m=+2.016921134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret") pod "global-pull-secret-syncer-btbxj" (UID: "ea88e142-7158-477d-b67a-1f1f450cbc6a") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931163 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-hostroot\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931182 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-hostroot\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931189 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ece76da-8177-4048-97cc-df65309482a6-system-cni-dir\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931227 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-etc-selinux\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931246 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkw8\" (UniqueName: \"kubernetes.io/projected/7ff6c377-123d-427c-9f57-748013cb3bcd-kube-api-access-zgkw8\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931246 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-device-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-multus-cni-dir\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931271 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-etc-selinux\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.931709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931307 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-kubernetes\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931332 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-run-systemd\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931338 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-multus-cni-dir\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-socket-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931375 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a1f2830-2236-4dab-bfed-a19ac01c8948-ovnkube-script-lib\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931399 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ece76da-8177-4048-97cc-df65309482a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931418 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-kubernetes\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931426 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ece76da-8177-4048-97cc-df65309482a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931450 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prmzl\" (UniqueName: \"kubernetes.io/projected/711caeef-4722-4a09-b5df-55262773ebb3-kube-api-access-prmzl\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931468 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-socket-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931480 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-run\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931499 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm8gk\" (UniqueName: \"kubernetes.io/projected/d7ce9c33-5241-41d5-8b0f-129237867308-kube-api-access-qm8gk\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931471 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/711caeef-4722-4a09-b5df-55262773ebb3-multus-daemon-config\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931514 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcqlk\" (UniqueName: \"kubernetes.io/projected/b01ecda8-8841-4584-a12b-daff7da47a47-kube-api-access-vcqlk\") pod \"node-resolver-9glj9\" (UID: \"b01ecda8-8841-4584-a12b-daff7da47a47\") " pod="openshift-dns/node-resolver-9glj9" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931577 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-run-netns\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931581 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-run\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931608 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc623d5a-e0b2-4aa5-967f-c8b417121518-serviceca\") pod \"node-ca-sxdzj\" (UID: \"dc623d5a-e0b2-4aa5-967f-c8b417121518\") " pod="openshift-image-registry/node-ca-sxdzj" Sep 16 01:15:50.932329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931634 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-run-k8s-cni-cncf-io\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931659 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-systemd-units\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931670 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-run-k8s-cni-cncf-io\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931686 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5lp5\" (UniqueName: \"kubernetes.io/projected/dc623d5a-e0b2-4aa5-967f-c8b417121518-kube-api-access-h5lp5\") pod \"node-ca-sxdzj\" (UID: \"dc623d5a-e0b2-4aa5-967f-c8b417121518\") " pod="openshift-image-registry/node-ca-sxdzj" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931709 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-var-lib-cni-bin\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931758 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-var-lib-cni-bin\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931829 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-multus-conf-dir\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931858 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-etc-kubernetes\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931891 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb26b3d5-1523-4e94-b72c-8a931b24e8a2-host-slash\") pod \"iptables-alerter-h92p2\" (UID: \"cb26b3d5-1523-4e94-b72c-8a931b24e8a2\") " pod="openshift-network-operator/iptables-alerter-h92p2" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931910 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-multus-conf-dir\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931944 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ce9c33-5241-41d5-8b0f-129237867308-tmp\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931953 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-etc-kubernetes\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931971 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-etc-openvswitch\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.931976 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb26b3d5-1523-4e94-b72c-8a931b24e8a2-host-slash\") pod \"iptables-alerter-h92p2\" (UID: \"cb26b3d5-1523-4e94-b72c-8a931b24e8a2\") " pod="openshift-network-operator/iptables-alerter-h92p2" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932010 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dc623d5a-e0b2-4aa5-967f-c8b417121518-serviceca\") pod \"node-ca-sxdzj\" (UID: \"dc623d5a-e0b2-4aa5-967f-c8b417121518\") " pod="openshift-image-registry/node-ca-sxdzj" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932539 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ece76da-8177-4048-97cc-df65309482a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932563 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-lib-modules\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932583 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea88e142-7158-477d-b67a-1f1f450cbc6a-dbus\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:50.932981 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932661 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ea88e142-7158-477d-b67a-1f1f450cbc6a-dbus\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932676 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-lib-modules\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932700 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c81a486-0ae1-4caa-be44-9a6cc0edf50c-agent-certs\") pod \"konnectivity-agent-qcv88\" (UID: \"5c81a486-0ae1-4caa-be44-9a6cc0edf50c\") " pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932719 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-systemd\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932735 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ece76da-8177-4048-97cc-df65309482a6-cnibin\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932762 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-host\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932786 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea88e142-7158-477d-b67a-1f1f450cbc6a-kubelet-config\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-systemd\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932808 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-registration-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932817 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-host\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932844 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-modprobe-d\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ff6c377-123d-427c-9f57-748013cb3bcd-registration-dir\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932872 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4mlz\" (UniqueName: \"kubernetes.io/projected/3ece76da-8177-4048-97cc-df65309482a6-kube-api-access-v4mlz\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932896 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c81a486-0ae1-4caa-be44-9a6cc0edf50c-konnectivity-ca\") pod \"konnectivity-agent-qcv88\" (UID: \"5c81a486-0ae1-4caa-be44-9a6cc0edf50c\") " pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932899 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ea88e142-7158-477d-b67a-1f1f450cbc6a-kubelet-config\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc623d5a-e0b2-4aa5-967f-c8b417121518-host\") pod \"node-ca-sxdzj\" (UID: \"dc623d5a-e0b2-4aa5-967f-c8b417121518\") " pod="openshift-image-registry/node-ca-sxdzj" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932925 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932942 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-system-cni-dir\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.933900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932962 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-cnibin\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932977 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/711caeef-4722-4a09-b5df-55262773ebb3-cni-binary-copy\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932944 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc623d5a-e0b2-4aa5-967f-c8b417121518-host\") pod \"node-ca-sxdzj\" (UID: \"dc623d5a-e0b2-4aa5-967f-c8b417121518\") " pod="openshift-image-registry/node-ca-sxdzj" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932991 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b01ecda8-8841-4584-a12b-daff7da47a47-tmp-dir\") pod \"node-resolver-9glj9\" (UID: \"b01ecda8-8841-4584-a12b-daff7da47a47\") " pod="openshift-dns/node-resolver-9glj9" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.932931 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-modprobe-d\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-run-ovn-kubernetes\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933032 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-system-cni-dir\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933059 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jszh\" (UniqueName: \"kubernetes.io/projected/cb26b3d5-1523-4e94-b72c-8a931b24e8a2-kube-api-access-9jszh\") pod \"iptables-alerter-h92p2\" (UID: \"cb26b3d5-1523-4e94-b72c-8a931b24e8a2\") " pod="openshift-network-operator/iptables-alerter-h92p2" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933092 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933120 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ece76da-8177-4048-97cc-df65309482a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933209 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-cnibin\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933252 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-var-lib-kubelet\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-sysctl-conf\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933336 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-var-lib-kubelet\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933374 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-var-lib-kubelet\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933381 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-host-var-lib-kubelet\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933389 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5c81a486-0ae1-4caa-be44-9a6cc0edf50c-konnectivity-ca\") pod \"konnectivity-agent-qcv88\" (UID: \"5c81a486-0ae1-4caa-be44-9a6cc0edf50c\") " pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:15:50.934964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933424 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-log-socket\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933448 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/711caeef-4722-4a09-b5df-55262773ebb3-cni-binary-copy\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933463 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d7ce9c33-5241-41d5-8b0f-129237867308-etc-tuned\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933464 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d7ce9c33-5241-41d5-8b0f-129237867308-etc-sysctl-conf\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933490 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b01ecda8-8841-4584-a12b-daff7da47a47-hosts-file\") pod \"node-resolver-9glj9\" (UID: \"b01ecda8-8841-4584-a12b-daff7da47a47\") " pod="openshift-dns/node-resolver-9glj9" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933514 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbdd\" (UniqueName: \"kubernetes.io/projected/00728728-c5dc-4f6c-b96b-8356103fafa5-kube-api-access-tnbdd\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933536 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-slash\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933557 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a1f2830-2236-4dab-bfed-a19ac01c8948-env-overrides\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933579 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rnf\" (UniqueName: \"kubernetes.io/projected/4a1f2830-2236-4dab-bfed-a19ac01c8948-kube-api-access-56rnf\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-multus-socket-dir-parent\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933632 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-run-ovn\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933653 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-node-log\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933673 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/711caeef-4722-4a09-b5df-55262773ebb3-multus-socket-dir-parent\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933674 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb26b3d5-1523-4e94-b72c-8a931b24e8a2-iptables-alerter-script\") pod \"iptables-alerter-h92p2\" (UID: \"cb26b3d5-1523-4e94-b72c-8a931b24e8a2\") " pod="openshift-network-operator/iptables-alerter-h92p2" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933718 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-kubelet\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933753 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-var-lib-openvswitch\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.933772 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a1f2830-2236-4dab-bfed-a19ac01c8948-ovnkube-config\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:50.935820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.934099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb26b3d5-1523-4e94-b72c-8a931b24e8a2-iptables-alerter-script\") pod \"iptables-alerter-h92p2\" (UID: \"cb26b3d5-1523-4e94-b72c-8a931b24e8a2\") " pod="openshift-network-operator/iptables-alerter-h92p2" Sep 16 01:15:50.936542 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.936314 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ce9c33-5241-41d5-8b0f-129237867308-tmp\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.936542 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.936351 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d7ce9c33-5241-41d5-8b0f-129237867308-etc-tuned\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.936542 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.936496 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5c81a486-0ae1-4caa-be44-9a6cc0edf50c-agent-certs\") pod \"konnectivity-agent-qcv88\" (UID: \"5c81a486-0ae1-4caa-be44-9a6cc0edf50c\") " pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:15:50.939709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.939678 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5lp5\" (UniqueName: \"kubernetes.io/projected/dc623d5a-e0b2-4aa5-967f-c8b417121518-kube-api-access-h5lp5\") pod \"node-ca-sxdzj\" (UID: \"dc623d5a-e0b2-4aa5-967f-c8b417121518\") " pod="openshift-image-registry/node-ca-sxdzj" Sep 16 01:15:50.939845 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.939827 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prmzl\" (UniqueName: \"kubernetes.io/projected/711caeef-4722-4a09-b5df-55262773ebb3-kube-api-access-prmzl\") pod \"multus-fxrml\" (UID: \"711caeef-4722-4a09-b5df-55262773ebb3\") " pod="openshift-multus/multus-fxrml" Sep 16 01:15:50.939975 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.939958 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkw8\" (UniqueName: \"kubernetes.io/projected/7ff6c377-123d-427c-9f57-748013cb3bcd-kube-api-access-zgkw8\") pod \"aws-ebs-csi-driver-node-nks55\" (UID: \"7ff6c377-123d-427c-9f57-748013cb3bcd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:50.940573 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.940548 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm8gk\" (UniqueName: \"kubernetes.io/projected/d7ce9c33-5241-41d5-8b0f-129237867308-kube-api-access-qm8gk\") pod \"tuned-s6rxq\" (UID: \"d7ce9c33-5241-41d5-8b0f-129237867308\") " pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:50.944012 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.943995 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jszh\" (UniqueName: \"kubernetes.io/projected/cb26b3d5-1523-4e94-b72c-8a931b24e8a2-kube-api-access-9jszh\") pod \"iptables-alerter-h92p2\" (UID: \"cb26b3d5-1523-4e94-b72c-8a931b24e8a2\") " pod="openshift-network-operator/iptables-alerter-h92p2" Sep 16 01:15:50.989232 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:50.989189 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee9021a69ef53dd07b6a5e501d5f066.slice/crio-be46a7405ab9a902d340ecbec526c1bb93ec1739646da41befdfe451b7aa1c8d WatchSource:0}: Error finding container be46a7405ab9a902d340ecbec526c1bb93ec1739646da41befdfe451b7aa1c8d: Status 404 returned error can't find the container with id be46a7405ab9a902d340ecbec526c1bb93ec1739646da41befdfe451b7aa1c8d Sep 16 01:15:50.989766 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:50.989747 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf880a3cf392188e514f562daba8d34c.slice/crio-ea6770e721da9d68c689186fac9fa89c187613603e754826aaed867a5d21271d WatchSource:0}: Error finding container ea6770e721da9d68c689186fac9fa89c187613603e754826aaed867a5d21271d: Status 404 returned error can't find the container with id ea6770e721da9d68c689186fac9fa89c187613603e754826aaed867a5d21271d Sep 16 01:15:50.995599 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:50.995584 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 01:15:51.034537 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034510 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-run-ovn\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034537 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034538 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-node-log\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034553 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-kubelet\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034567 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-var-lib-openvswitch\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034587 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a1f2830-2236-4dab-bfed-a19ac01c8948-ovnkube-config\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034610 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034622 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-node-log\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034634 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-run-ovn\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034648 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-kubelet\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034674 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-var-lib-openvswitch\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034666 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-run-openvswitch\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034698 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-run-openvswitch\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034728 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a1f2830-2236-4dab-bfed-a19ac01c8948-ovn-node-metrics-cert\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.034745 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034782 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf9f\" (UniqueName: \"kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f\") pod \"network-check-target-xgkwn\" (UID: \"a8146604-b0d8-41a2-bd14-3daff2e45882\") " pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:15:51.034798 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.034806 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs podName:00728728-c5dc-4f6c-b96b-8356103fafa5 nodeName:}" failed. No retries permitted until 2026-09-16 01:15:51.534787029 +0000 UTC m=+2.120589219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs") pod "network-metrics-daemon-snt2b" (UID: "00728728-c5dc-4f6c-b96b-8356103fafa5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034835 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-cni-bin\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034861 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-cni-netd\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034890 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ece76da-8177-4048-97cc-df65309482a6-system-cni-dir\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034894 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-cni-bin\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034919 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-run-systemd\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034945 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a1f2830-2236-4dab-bfed-a19ac01c8948-ovnkube-script-lib\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034939 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-cni-netd\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034975 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3ece76da-8177-4048-97cc-df65309482a6-system-cni-dir\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.034976 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ece76da-8177-4048-97cc-df65309482a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035013 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-run-systemd\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035020 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ece76da-8177-4048-97cc-df65309482a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035050 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcqlk\" (UniqueName: \"kubernetes.io/projected/b01ecda8-8841-4584-a12b-daff7da47a47-kube-api-access-vcqlk\") pod \"node-resolver-9glj9\" (UID: \"b01ecda8-8841-4584-a12b-daff7da47a47\") " pod="openshift-dns/node-resolver-9glj9" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035072 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-run-netns\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035092 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-systemd-units\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035124 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-etc-openvswitch\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035145 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-run-netns\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.035462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035150 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ece76da-8177-4048-97cc-df65309482a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035195 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ece76da-8177-4048-97cc-df65309482a6-cnibin\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035226 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4mlz\" (UniqueName: \"kubernetes.io/projected/3ece76da-8177-4048-97cc-df65309482a6-kube-api-access-v4mlz\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035264 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b01ecda8-8841-4584-a12b-daff7da47a47-tmp-dir\") pod \"node-resolver-9glj9\" (UID: \"b01ecda8-8841-4584-a12b-daff7da47a47\") " pod="openshift-dns/node-resolver-9glj9" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-run-ovn-kubernetes\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035324 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a1f2830-2236-4dab-bfed-a19ac01c8948-ovnkube-config\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035336 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035362 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ece76da-8177-4048-97cc-df65309482a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035394 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-log-socket\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035363 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-systemd-units\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035418 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b01ecda8-8841-4584-a12b-daff7da47a47-hosts-file\") pod \"node-resolver-9glj9\" (UID: \"b01ecda8-8841-4584-a12b-daff7da47a47\") " pod="openshift-dns/node-resolver-9glj9" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035404 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035472 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbdd\" (UniqueName: \"kubernetes.io/projected/00728728-c5dc-4f6c-b96b-8356103fafa5-kube-api-access-tnbdd\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035483 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-log-socket\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035496 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3ece76da-8177-4048-97cc-df65309482a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035392 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-etc-openvswitch\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035497 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-slash\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.036435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035535 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a1f2830-2236-4dab-bfed-a19ac01c8948-env-overrides\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.037199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035540 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3ece76da-8177-4048-97cc-df65309482a6-cnibin\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.037199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035552 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a1f2830-2236-4dab-bfed-a19ac01c8948-ovnkube-script-lib\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.037199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rnf\" (UniqueName: \"kubernetes.io/projected/4a1f2830-2236-4dab-bfed-a19ac01c8948-kube-api-access-56rnf\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.037199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035600 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3ece76da-8177-4048-97cc-df65309482a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.037199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035610 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b01ecda8-8841-4584-a12b-daff7da47a47-tmp-dir\") pod \"node-resolver-9glj9\" (UID: \"b01ecda8-8841-4584-a12b-daff7da47a47\") " pod="openshift-dns/node-resolver-9glj9" Sep 16 01:15:51.037199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035648 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-run-ovn-kubernetes\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.037199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035892 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b01ecda8-8841-4584-a12b-daff7da47a47-hosts-file\") pod \"node-resolver-9glj9\" (UID: \"b01ecda8-8841-4584-a12b-daff7da47a47\") " pod="openshift-dns/node-resolver-9glj9" Sep 16 01:15:51.037199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.036052 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3ece76da-8177-4048-97cc-df65309482a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.037199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.036079 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3ece76da-8177-4048-97cc-df65309482a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.037199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.036118 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a1f2830-2236-4dab-bfed-a19ac01c8948-env-overrides\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.037199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.035578 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a1f2830-2236-4dab-bfed-a19ac01c8948-host-slash\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.037812 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.037797 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a1f2830-2236-4dab-bfed-a19ac01c8948-ovn-node-metrics-cert\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.050396 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.048238 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:15:51.050396 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.048321 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:15:51.050396 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.048337 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vf9f for pod openshift-network-diagnostics/network-check-target-xgkwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:15:51.050396 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.048419 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f podName:a8146604-b0d8-41a2-bd14-3daff2e45882 nodeName:}" failed. No retries permitted until 2026-09-16 01:15:51.548401041 +0000 UTC m=+2.134203230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7vf9f" (UniqueName: "kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f") pod "network-check-target-xgkwn" (UID: "a8146604-b0d8-41a2-bd14-3daff2e45882") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:15:51.050866 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.050808 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rnf\" (UniqueName: \"kubernetes.io/projected/4a1f2830-2236-4dab-bfed-a19ac01c8948-kube-api-access-56rnf\") pod \"ovnkube-node-79tjp\" (UID: \"4a1f2830-2236-4dab-bfed-a19ac01c8948\") " pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.051138 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.051118 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4mlz\" (UniqueName: \"kubernetes.io/projected/3ece76da-8177-4048-97cc-df65309482a6-kube-api-access-v4mlz\") pod \"multus-additional-cni-plugins-78x9p\" (UID: \"3ece76da-8177-4048-97cc-df65309482a6\") " pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.052176 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.052156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcqlk\" (UniqueName: \"kubernetes.io/projected/b01ecda8-8841-4584-a12b-daff7da47a47-kube-api-access-vcqlk\") pod \"node-resolver-9glj9\" (UID: \"b01ecda8-8841-4584-a12b-daff7da47a47\") " pod="openshift-dns/node-resolver-9glj9" Sep 16 01:15:51.053268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.053247 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbdd\" (UniqueName: \"kubernetes.io/projected/00728728-c5dc-4f6c-b96b-8356103fafa5-kube-api-access-tnbdd\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:51.131187 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.131110 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:15:51.142002 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:51.141979 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c81a486_0ae1_4caa_be44_9a6cc0edf50c.slice/crio-fdc24a3ba9eac6b97e790fa2067c899573711067c346cd948c5e5aa1f9d6db7a WatchSource:0}: Error finding container fdc24a3ba9eac6b97e790fa2067c899573711067c346cd948c5e5aa1f9d6db7a: Status 404 returned error can't find the container with id fdc24a3ba9eac6b97e790fa2067c899573711067c346cd948c5e5aa1f9d6db7a Sep 16 01:15:51.143793 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.143769 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" Sep 16 01:15:51.152342 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:51.152321 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff6c377_123d_427c_9f57_748013cb3bcd.slice/crio-c8dff624e446b72fc99a6c3d2a5f52553ff69afd3f74723995d4aeb654e0f177 WatchSource:0}: Error finding container c8dff624e446b72fc99a6c3d2a5f52553ff69afd3f74723995d4aeb654e0f177: Status 404 returned error can't find the container with id c8dff624e446b72fc99a6c3d2a5f52553ff69afd3f74723995d4aeb654e0f177 Sep 16 01:15:51.161137 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.161115 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sxdzj" Sep 16 01:15:51.166395 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.166376 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fxrml" Sep 16 01:15:51.170429 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:51.170408 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc623d5a_e0b2_4aa5_967f_c8b417121518.slice/crio-207809b93d3eec90b3163889143ce357cabcc804eefd707973c02c28f8b4ee87 WatchSource:0}: Error finding container 207809b93d3eec90b3163889143ce357cabcc804eefd707973c02c28f8b4ee87: Status 404 returned error can't find the container with id 207809b93d3eec90b3163889143ce357cabcc804eefd707973c02c28f8b4ee87 Sep 16 01:15:51.174961 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:51.174939 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711caeef_4722_4a09_b5df_55262773ebb3.slice/crio-892de8e3c61004dfc7325f8efde783e25b52cfd1969268448ef52829fb043c11 WatchSource:0}: Error finding container 892de8e3c61004dfc7325f8efde783e25b52cfd1969268448ef52829fb043c11: Status 404 returned error can't find the container with id 892de8e3c61004dfc7325f8efde783e25b52cfd1969268448ef52829fb043c11 Sep 16 01:15:51.181293 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.181275 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h92p2" Sep 16 01:15:51.191993 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:51.191973 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb26b3d5_1523_4e94_b72c_8a931b24e8a2.slice/crio-80ff1d2f68436a3d3ac5f2e99e773d66a615a37fb529f9b1abf698fd9a949f6b WatchSource:0}: Error finding container 80ff1d2f68436a3d3ac5f2e99e773d66a615a37fb529f9b1abf698fd9a949f6b: Status 404 returned error can't find the container with id 80ff1d2f68436a3d3ac5f2e99e773d66a615a37fb529f9b1abf698fd9a949f6b Sep 16 01:15:51.198158 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.198139 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" Sep 16 01:15:51.206456 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:51.206435 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ce9c33_5241_41d5_8b0f_129237867308.slice/crio-76b1cbc0d6e329e872349e6a34429a410ed26d401af63e9d53f1e106299358c7 WatchSource:0}: Error finding container 76b1cbc0d6e329e872349e6a34429a410ed26d401af63e9d53f1e106299358c7: Status 404 returned error can't find the container with id 76b1cbc0d6e329e872349e6a34429a410ed26d401af63e9d53f1e106299358c7 Sep 16 01:15:51.213032 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.213010 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9glj9" Sep 16 01:15:51.221980 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:51.221954 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01ecda8_8841_4584_a12b_daff7da47a47.slice/crio-9ab0ff5ed45cc660c2b35955dd84c5686fae8f54f252461f5bca5cff6a51ac67 WatchSource:0}: Error finding container 9ab0ff5ed45cc660c2b35955dd84c5686fae8f54f252461f5bca5cff6a51ac67: Status 404 returned error can't find the container with id 9ab0ff5ed45cc660c2b35955dd84c5686fae8f54f252461f5bca5cff6a51ac67 Sep 16 01:15:51.226586 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.226564 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-78x9p" Sep 16 01:15:51.231834 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.231811 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:15:51.238188 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:51.238167 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ece76da_8177_4048_97cc_df65309482a6.slice/crio-62ba6785b3a8fe6f367ca70bfca568bfc7d3f541447117584019f43fb4b04a05 WatchSource:0}: Error finding container 62ba6785b3a8fe6f367ca70bfca568bfc7d3f541447117584019f43fb4b04a05: Status 404 returned error can't find the container with id 62ba6785b3a8fe6f367ca70bfca568bfc7d3f541447117584019f43fb4b04a05 Sep 16 01:15:51.241751 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:15:51.241732 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1f2830_2236_4dab_bfed_a19ac01c8948.slice/crio-0581012823fce535d28dda62c8ff358c434edd1c6c3ed53f80b7e096e46163e4 WatchSource:0}: Error finding container 0581012823fce535d28dda62c8ff358c434edd1c6c3ed53f80b7e096e46163e4: Status 404 returned error can't find the container with id 0581012823fce535d28dda62c8ff358c434edd1c6c3ed53f80b7e096e46163e4 Sep 16 01:15:51.438801 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.438714 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:51.438973 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.438882 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:15:51.438973 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.438950 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret podName:ea88e142-7158-477d-b67a-1f1f450cbc6a nodeName:}" failed. No retries permitted until 2026-09-16 01:15:52.43893404 +0000 UTC m=+3.024736230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret") pod "global-pull-secret-syncer-btbxj" (UID: "ea88e142-7158-477d-b67a-1f1f450cbc6a") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:15:51.539905 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.539870 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:51.540098 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.540021 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:15:51.540098 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.540091 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs podName:00728728-c5dc-4f6c-b96b-8356103fafa5 nodeName:}" failed. No retries permitted until 2026-09-16 01:15:52.540072826 +0000 UTC m=+3.125875036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs") pod "network-metrics-daemon-snt2b" (UID: "00728728-c5dc-4f6c-b96b-8356103fafa5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:15:51.640582 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.640540 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf9f\" (UniqueName: \"kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f\") pod \"network-check-target-xgkwn\" (UID: \"a8146604-b0d8-41a2-bd14-3daff2e45882\") " pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:15:51.640878 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.640853 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:15:51.640972 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.640888 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:15:51.640972 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.640902 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vf9f for pod openshift-network-diagnostics/network-check-target-xgkwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:15:51.640972 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.640963 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f podName:a8146604-b0d8-41a2-bd14-3daff2e45882 nodeName:}" failed. No retries permitted until 2026-09-16 01:15:52.640943221 +0000 UTC m=+3.226745427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vf9f" (UniqueName: "kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f") pod "network-check-target-xgkwn" (UID: "a8146604-b0d8-41a2-bd14-3daff2e45882") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:15:51.865161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.865037 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 01:10:51 +0000 UTC" deadline="2028-06-03 13:38:43.816522056 +0000 UTC" Sep 16 01:15:51.865161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.865116 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15036h22m51.951410657s" Sep 16 01:15:51.922166 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.922128 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 01:15:51.936532 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.936502 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:51.936713 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:51.936650 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:15:51.947733 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.947610 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" event={"ID":"4ee9021a69ef53dd07b6a5e501d5f066","Type":"ContainerStarted","Data":"be46a7405ab9a902d340ecbec526c1bb93ec1739646da41befdfe451b7aa1c8d"} Sep 16 01:15:51.950779 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.950745 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-56.ec2.internal" event={"ID":"af880a3cf392188e514f562daba8d34c","Type":"ContainerStarted","Data":"ea6770e721da9d68c689186fac9fa89c187613603e754826aaed867a5d21271d"} Sep 16 01:15:51.952900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.952870 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" event={"ID":"d7ce9c33-5241-41d5-8b0f-129237867308","Type":"ContainerStarted","Data":"76b1cbc0d6e329e872349e6a34429a410ed26d401af63e9d53f1e106299358c7"} Sep 16 01:15:51.957518 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.957491 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fxrml" event={"ID":"711caeef-4722-4a09-b5df-55262773ebb3","Type":"ContainerStarted","Data":"892de8e3c61004dfc7325f8efde783e25b52cfd1969268448ef52829fb043c11"} Sep 16 01:15:51.964536 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.964508 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qcv88" event={"ID":"5c81a486-0ae1-4caa-be44-9a6cc0edf50c","Type":"ContainerStarted","Data":"fdc24a3ba9eac6b97e790fa2067c899573711067c346cd948c5e5aa1f9d6db7a"} Sep 16 01:15:51.984489 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.984303 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" event={"ID":"4a1f2830-2236-4dab-bfed-a19ac01c8948","Type":"ContainerStarted","Data":"0581012823fce535d28dda62c8ff358c434edd1c6c3ed53f80b7e096e46163e4"} Sep 16 01:15:51.996693 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.996642 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78x9p" event={"ID":"3ece76da-8177-4048-97cc-df65309482a6","Type":"ContainerStarted","Data":"62ba6785b3a8fe6f367ca70bfca568bfc7d3f541447117584019f43fb4b04a05"} Sep 16 01:15:51.999804 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:51.999686 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9glj9" event={"ID":"b01ecda8-8841-4584-a12b-daff7da47a47","Type":"ContainerStarted","Data":"9ab0ff5ed45cc660c2b35955dd84c5686fae8f54f252461f5bca5cff6a51ac67"} Sep 16 01:15:52.002760 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:52.002734 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h92p2" event={"ID":"cb26b3d5-1523-4e94-b72c-8a931b24e8a2","Type":"ContainerStarted","Data":"80ff1d2f68436a3d3ac5f2e99e773d66a615a37fb529f9b1abf698fd9a949f6b"} Sep 16 01:15:52.017958 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:52.017925 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sxdzj" event={"ID":"dc623d5a-e0b2-4aa5-967f-c8b417121518","Type":"ContainerStarted","Data":"207809b93d3eec90b3163889143ce357cabcc804eefd707973c02c28f8b4ee87"} Sep 16 01:15:52.024547 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:52.024518 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" event={"ID":"7ff6c377-123d-427c-9f57-748013cb3bcd","Type":"ContainerStarted","Data":"c8dff624e446b72fc99a6c3d2a5f52553ff69afd3f74723995d4aeb654e0f177"} Sep 16 01:15:52.448239 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:52.448158 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:52.448561 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:52.448405 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:15:52.448561 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:52.448478 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret podName:ea88e142-7158-477d-b67a-1f1f450cbc6a nodeName:}" failed. No retries permitted until 2026-09-16 01:15:54.448458135 +0000 UTC m=+5.034260354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret") pod "global-pull-secret-syncer-btbxj" (UID: "ea88e142-7158-477d-b67a-1f1f450cbc6a") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:15:52.549499 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:52.549454 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:52.549671 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:52.549648 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:15:52.549752 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:52.549735 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs podName:00728728-c5dc-4f6c-b96b-8356103fafa5 nodeName:}" failed. No retries permitted until 2026-09-16 01:15:54.549697131 +0000 UTC m=+5.135499346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs") pod "network-metrics-daemon-snt2b" (UID: "00728728-c5dc-4f6c-b96b-8356103fafa5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:15:52.650849 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:52.650592 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf9f\" (UniqueName: \"kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f\") pod \"network-check-target-xgkwn\" (UID: \"a8146604-b0d8-41a2-bd14-3daff2e45882\") " pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:15:52.650849 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:52.650841 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:15:52.651092 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:52.650871 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:15:52.651092 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:52.650885 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vf9f for pod openshift-network-diagnostics/network-check-target-xgkwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:15:52.651092 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:52.650950 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f podName:a8146604-b0d8-41a2-bd14-3daff2e45882 nodeName:}" failed. No retries permitted until 2026-09-16 01:15:54.650930928 +0000 UTC m=+5.236733124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vf9f" (UniqueName: "kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f") pod "network-check-target-xgkwn" (UID: "a8146604-b0d8-41a2-bd14-3daff2e45882") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:15:52.865635 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:52.865524 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 01:10:51 +0000 UTC" deadline="2028-04-02 12:41:57.925958023 +0000 UTC" Sep 16 01:15:52.865635 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:52.865573 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13547h26m5.060392228s" Sep 16 01:15:52.934004 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:52.933767 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:15:52.934247 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:52.934026 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:52.934247 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:52.934116 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:15:52.934455 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:52.934393 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:15:53.940238 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:53.940106 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:53.940699 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:53.940242 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:15:54.466540 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:54.466499 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:54.466743 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:54.466671 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:15:54.466743 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:54.466738 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret podName:ea88e142-7158-477d-b67a-1f1f450cbc6a nodeName:}" failed. No retries permitted until 2026-09-16 01:15:58.466721022 +0000 UTC m=+9.052523226 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret") pod "global-pull-secret-syncer-btbxj" (UID: "ea88e142-7158-477d-b67a-1f1f450cbc6a") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:15:54.568042 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:54.567746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:54.568042 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:54.567967 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:15:54.568042 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:54.568037 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs podName:00728728-c5dc-4f6c-b96b-8356103fafa5 nodeName:}" failed. No retries permitted until 2026-09-16 01:15:58.568012587 +0000 UTC m=+9.153814792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs") pod "network-metrics-daemon-snt2b" (UID: "00728728-c5dc-4f6c-b96b-8356103fafa5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:15:54.668376 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:54.668340 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf9f\" (UniqueName: \"kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f\") pod \"network-check-target-xgkwn\" (UID: \"a8146604-b0d8-41a2-bd14-3daff2e45882\") " pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:15:54.668556 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:54.668545 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:15:54.668614 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:54.668570 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:15:54.668614 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:54.668583 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vf9f for pod openshift-network-diagnostics/network-check-target-xgkwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:15:54.668720 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:54.668640 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f podName:a8146604-b0d8-41a2-bd14-3daff2e45882 nodeName:}" failed. No retries permitted until 2026-09-16 01:15:58.668622518 +0000 UTC m=+9.254424721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vf9f" (UniqueName: "kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f") pod "network-check-target-xgkwn" (UID: "a8146604-b0d8-41a2-bd14-3daff2e45882") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:15:54.933991 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:54.933884 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:54.934138 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:54.934017 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:15:54.934418 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:54.933883 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:15:54.934517 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:54.934495 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:15:55.934556 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:55.934508 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:55.934993 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:55.934744 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:15:56.933510 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:56.933472 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:56.933701 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:56.933602 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:15:56.933970 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:56.933946 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:15:56.934061 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:56.934034 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:15:57.933057 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:57.933025 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:57.933510 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:57.933184 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:15:58.500387 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:58.500290 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:58.500530 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:58.500410 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:15:58.500530 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:58.500478 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret podName:ea88e142-7158-477d-b67a-1f1f450cbc6a nodeName:}" failed. No retries permitted until 2026-09-16 01:16:06.500462258 +0000 UTC m=+17.086264451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret") pod "global-pull-secret-syncer-btbxj" (UID: "ea88e142-7158-477d-b67a-1f1f450cbc6a") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:15:58.601987 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:58.601425 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:58.601987 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:58.601597 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:15:58.601987 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:58.601659 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs podName:00728728-c5dc-4f6c-b96b-8356103fafa5 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:06.601641145 +0000 UTC m=+17.187443348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs") pod "network-metrics-daemon-snt2b" (UID: "00728728-c5dc-4f6c-b96b-8356103fafa5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:15:58.702113 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:58.702029 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf9f\" (UniqueName: \"kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f\") pod \"network-check-target-xgkwn\" (UID: \"a8146604-b0d8-41a2-bd14-3daff2e45882\") " pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:15:58.702311 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:58.702226 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:15:58.702311 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:58.702251 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:15:58.702311 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:58.702264 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vf9f for pod openshift-network-diagnostics/network-check-target-xgkwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:15:58.702475 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:58.702336 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f podName:a8146604-b0d8-41a2-bd14-3daff2e45882 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:06.702321101 +0000 UTC m=+17.288123294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vf9f" (UniqueName: "kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f") pod "network-check-target-xgkwn" (UID: "a8146604-b0d8-41a2-bd14-3daff2e45882") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:15:58.933753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:58.933721 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:15:58.934201 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:58.933837 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:15:58.934300 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:58.934209 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:15:58.934410 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:58.934371 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:15:59.934419 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:15:59.934384 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:15:59.934863 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:15:59.934498 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:00.933000 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:00.932954 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:00.933181 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:00.933015 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:00.933181 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:00.933113 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:00.933316 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:00.933246 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:01.933242 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:01.932959 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:01.933242 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:01.933125 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:02.933668 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:02.933627 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:02.933668 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:02.933658 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:02.934063 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:02.933750 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:02.934063 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:02.933832 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:03.933381 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:03.933345 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:03.933554 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:03.933479 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:04.933704 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:04.933667 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:04.934251 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:04.933668 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:04.934251 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:04.933804 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:04.934251 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:04.933863 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:05.935784 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:05.935748 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:05.936232 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:05.935873 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:06.563818 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:06.563769 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:06.564011 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:06.563919 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:06.564011 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:06.563981 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret podName:ea88e142-7158-477d-b67a-1f1f450cbc6a nodeName:}" failed. No retries permitted until 2026-09-16 01:16:22.563964961 +0000 UTC m=+33.149767163 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret") pod "global-pull-secret-syncer-btbxj" (UID: "ea88e142-7158-477d-b67a-1f1f450cbc6a") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:06.665063 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:06.665033 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:06.665241 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:06.665194 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:06.665316 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:06.665258 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs podName:00728728-c5dc-4f6c-b96b-8356103fafa5 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:22.665244561 +0000 UTC m=+33.251046749 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs") pod "network-metrics-daemon-snt2b" (UID: "00728728-c5dc-4f6c-b96b-8356103fafa5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:06.765435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:06.765399 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf9f\" (UniqueName: \"kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f\") pod \"network-check-target-xgkwn\" (UID: \"a8146604-b0d8-41a2-bd14-3daff2e45882\") " pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:06.765666 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:06.765524 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:16:06.765666 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:06.765547 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:16:06.765666 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:06.765560 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vf9f for pod openshift-network-diagnostics/network-check-target-xgkwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:06.765666 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:06.765622 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f podName:a8146604-b0d8-41a2-bd14-3daff2e45882 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:22.765603493 +0000 UTC m=+33.351405681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vf9f" (UniqueName: "kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f") pod "network-check-target-xgkwn" (UID: "a8146604-b0d8-41a2-bd14-3daff2e45882") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:06.933688 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:06.933605 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:06.933688 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:06.933645 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:06.933929 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:06.933722 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:06.933929 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:06.933852 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:07.933715 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:07.933676 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:07.934183 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:07.933815 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:08.933767 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:08.933727 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:08.934159 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:08.933776 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:08.934159 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:08.933873 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:08.934159 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:08.934004 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:09.934377 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:09.934343 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:09.934837 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:09.934444 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:10.933059 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:10.932831 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:10.933202 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:10.933177 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:10.933518 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:10.933358 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:10.933518 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:10.933458 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:11.090219 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:11.090114 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" event={"ID":"d7ce9c33-5241-41d5-8b0f-129237867308","Type":"ContainerStarted","Data":"ba3388e07c3ae1d6ceaa66b5e18807d3aca2dcb878e8d5c4c3297ff8d5613358"} Sep 16 01:16:11.093418 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:11.092986 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fxrml" event={"ID":"711caeef-4722-4a09-b5df-55262773ebb3","Type":"ContainerStarted","Data":"4da54b1fd78fe3037eda2c0cc7021ec3deba8d92e537c5f19f7b02efb1e6f4ef"} Sep 16 01:16:11.096220 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:11.096196 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" event={"ID":"4a1f2830-2236-4dab-bfed-a19ac01c8948","Type":"ContainerStarted","Data":"f45f6b6ecad85b02e4c2706c06b644b8995eb65c581d80dc98cbb5c80bd73222"} Sep 16 01:16:11.096336 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:11.096254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" event={"ID":"4a1f2830-2236-4dab-bfed-a19ac01c8948","Type":"ContainerStarted","Data":"49e95766743acd7be1fb56fe8456c0e34449c60f45d416424aafe21ce705bef0"} Sep 16 01:16:11.096572 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:11.096552 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" event={"ID":"4a1f2830-2236-4dab-bfed-a19ac01c8948","Type":"ContainerStarted","Data":"18a16bdee4a81ed70f0f27f6f49d06ead6395526b0e7467a3820a4f293860fc7"} Sep 16 01:16:11.098094 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:11.097799 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-56.ec2.internal" event={"ID":"af880a3cf392188e514f562daba8d34c","Type":"ContainerStarted","Data":"5f8ddf2f4cadd568b84611ec28be0d027386c20de2d9dd20b48ad576d77bf399"} Sep 16 01:16:11.127181 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:11.125949 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fxrml" podStartSLOduration=1.611862895 podStartE2EDuration="21.125927797s" totalImagesPullingTime="19.514064902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:11.12471438 +0000 UTC m=+21.710516592" watchObservedRunningTime="2026-09-16 01:16:11.125927797 +0000 UTC m=+21.711730009" Sep 16 01:16:11.127181 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:11.126481 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s6rxq" podStartSLOduration=1.673387382 podStartE2EDuration="21.126465153s" totalImagesPullingTime="19.453077771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:11.108349462 +0000 UTC m=+21.694151676" watchObservedRunningTime="2026-09-16 01:16:11.126465153 +0000 UTC m=+21.712267365" Sep 16 01:16:11.142873 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:11.142818 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-56.ec2.internal" podStartSLOduration=21.142801878 podStartE2EDuration="21.142801878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:11.142463385 +0000 UTC m=+21.728265597" watchObservedRunningTime="2026-09-16 01:16:11.142801878 +0000 UTC m=+21.728604088" Sep 16 01:16:11.933216 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:11.933185 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:11.933455 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:11.933330 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:12.102685 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.102654 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" event={"ID":"4a1f2830-2236-4dab-bfed-a19ac01c8948","Type":"ContainerStarted","Data":"bbed1e9a432b84cdafd5a082d8803054c0ee790095ca0e63991af805e9d7b0cb"} Sep 16 01:16:12.103155 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.102729 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" event={"ID":"4a1f2830-2236-4dab-bfed-a19ac01c8948","Type":"ContainerStarted","Data":"b9df413a8fbf59ec813cab3e7fcc66158f638fe119d66112d5dd856e58411846"} Sep 16 01:16:12.103155 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.102766 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" event={"ID":"4a1f2830-2236-4dab-bfed-a19ac01c8948","Type":"ContainerStarted","Data":"f8d8f7fd5cecf1f459203a30cd361be6b9f278e2c0bb8d033cbe2a99a4513e9e"} Sep 16 01:16:12.104053 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.104032 2567 generic.go:356] "Generic (PLEG): container finished" podID="3ece76da-8177-4048-97cc-df65309482a6" containerID="5e3c0fb4ac5e9ca1dbba40caf5e3f3b0ad16020b8b3f065e592fef33a9a16e2d" exitCode=0 Sep 16 01:16:12.104176 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.104110 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78x9p" event={"ID":"3ece76da-8177-4048-97cc-df65309482a6","Type":"ContainerDied","Data":"5e3c0fb4ac5e9ca1dbba40caf5e3f3b0ad16020b8b3f065e592fef33a9a16e2d"} Sep 16 01:16:12.105760 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.105737 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9glj9" event={"ID":"b01ecda8-8841-4584-a12b-daff7da47a47","Type":"ContainerStarted","Data":"8582337e6b4ca8ccafdaca7fa61d3b147104ccbbe67510f5249ebb468f0bd4c2"} Sep 16 01:16:12.111660 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.111624 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sxdzj" event={"ID":"dc623d5a-e0b2-4aa5-967f-c8b417121518","Type":"ContainerStarted","Data":"104c00959b2a12d713e2d295f34c6bc4810ec30066004834c6d1ab3665766705"} Sep 16 01:16:12.113153 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.113130 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" event={"ID":"7ff6c377-123d-427c-9f57-748013cb3bcd","Type":"ContainerStarted","Data":"9375ffa40954bd8bb807cbfc39737595d04ea63d1db21748a472f625dec26cd7"} Sep 16 01:16:12.114527 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.114504 2567 generic.go:356] "Generic (PLEG): container finished" podID="4ee9021a69ef53dd07b6a5e501d5f066" containerID="073353e8622b617e2a9a18f7dfa85256a8557b7b546f5376d6b079b2465f4e57" exitCode=0 Sep 16 01:16:12.114602 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.114569 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" event={"ID":"4ee9021a69ef53dd07b6a5e501d5f066","Type":"ContainerDied","Data":"073353e8622b617e2a9a18f7dfa85256a8557b7b546f5376d6b079b2465f4e57"} Sep 16 01:16:12.115800 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.115779 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qcv88" event={"ID":"5c81a486-0ae1-4caa-be44-9a6cc0edf50c","Type":"ContainerStarted","Data":"6a45d47896499ed8b207f3e0edadf6eaaea84a612b37e125bb6140237d4d9751"} Sep 16 01:16:12.174019 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.173966 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qcv88" podStartSLOduration=2.655665077 podStartE2EDuration="22.17395013s" totalImagesPullingTime="19.518285053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:12.17377454 +0000 UTC m=+22.759576751" watchObservedRunningTime="2026-09-16 01:16:12.17395013 +0000 UTC m=+22.759752340" Sep 16 01:16:12.174485 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.174422 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9glj9" podStartSLOduration=2.736379146 podStartE2EDuration="22.174414872s" totalImagesPullingTime="19.438035726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:12.155488338 +0000 UTC m=+22.741290549" watchObservedRunningTime="2026-09-16 01:16:12.174414872 +0000 UTC m=+22.760217083" Sep 16 01:16:12.190162 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.190122 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sxdzj" podStartSLOduration=2.7003792620000002 podStartE2EDuration="22.190104069s" totalImagesPullingTime="19.489724807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:12.190000498 +0000 UTC m=+22.775802710" watchObservedRunningTime="2026-09-16 01:16:12.190104069 +0000 UTC m=+22.775906281" Sep 16 01:16:12.712602 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.712574 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 01:16:12.890883 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.890687 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T01:16:12.712598344Z","UUID":"8d510676-e6b2-45d8-a3b3-a9cc9cd0b72f","Handler":null,"Name":"","Endpoint":""} Sep 16 01:16:12.892173 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.892151 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 01:16:12.892334 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.892186 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 01:16:12.933400 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.933372 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:12.933400 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:12.933391 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:12.933639 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:12.933546 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:12.933698 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:12.933646 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:13.122050 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:13.122016 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h92p2" event={"ID":"cb26b3d5-1523-4e94-b72c-8a931b24e8a2","Type":"ContainerStarted","Data":"f042e400cd17afd4e4d925509e8f08c0bd7437d5ff187b920980aa6e78f7d8cb"} Sep 16 01:16:13.124137 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:13.124107 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" event={"ID":"7ff6c377-123d-427c-9f57-748013cb3bcd","Type":"ContainerStarted","Data":"51a2c66004515176fce3077b3bec274d8336f73c3279b50d975f46861c9b499d"} Sep 16 01:16:13.126161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:13.126134 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" event={"ID":"4ee9021a69ef53dd07b6a5e501d5f066","Type":"ContainerStarted","Data":"919c926990bbac53d770dd561f4ef66570b4171f0f2cbeba6210c2aecf0dda0a"} Sep 16 01:16:13.148330 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:13.148229 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-56.ec2.internal" podStartSLOduration=23.148214775 podStartE2EDuration="23.148214775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:13.147763351 +0000 UTC m=+23.733565573" watchObservedRunningTime="2026-09-16 01:16:13.148214775 +0000 UTC m=+23.734016985" Sep 16 01:16:13.148473 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:13.148367 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h92p2" podStartSLOduration=3.680158709 podStartE2EDuration="23.148362088s" totalImagesPullingTime="19.468203379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:13.135526106 +0000 UTC m=+23.721328317" watchObservedRunningTime="2026-09-16 01:16:13.148362088 +0000 UTC m=+23.734164298" Sep 16 01:16:13.696656 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:13.696614 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:16:13.697456 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:13.697431 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:16:13.933110 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:13.933076 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:13.933298 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:13.933216 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:14.132441 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:14.132405 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" event={"ID":"4a1f2830-2236-4dab-bfed-a19ac01c8948","Type":"ContainerStarted","Data":"aee620e0c5c5b389d4dd6e245dcf81abd30715789d7aafe30a57c4aae7d7db9b"} Sep 16 01:16:14.133060 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:14.133035 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:16:14.133366 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:14.133348 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qcv88" Sep 16 01:16:14.933583 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:14.933416 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:14.933787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:14.933417 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:14.933787 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:14.933651 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:14.933787 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:14.933730 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:15.137027 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:15.136991 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" event={"ID":"7ff6c377-123d-427c-9f57-748013cb3bcd","Type":"ContainerStarted","Data":"515c42d2720a645d523d86516ea3b134004a1e1aeb2ec0c4865e45fd223dfbdc"} Sep 16 01:16:15.155416 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:15.155364 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nks55" podStartSLOduration=2.874204577 podStartE2EDuration="25.15534667s" totalImagesPullingTime="22.281142093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:15.153534297 +0000 UTC m=+25.739336504" watchObservedRunningTime="2026-09-16 01:16:15.15534667 +0000 UTC m=+25.741148881" Sep 16 01:16:15.933771 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:15.933736 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:15.933936 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:15.933882 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:16.933789 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:16.933758 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:16.934194 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:16.933778 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:16.934194 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:16.933870 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:16.934194 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:16.933969 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:17.146518 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:17.146482 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" event={"ID":"4a1f2830-2236-4dab-bfed-a19ac01c8948","Type":"ContainerStarted","Data":"25d00a67d9cb3bbd3044d2b1d518750118ec10551fd4bb47e922da9da69d7b82"} Sep 16 01:16:17.146820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:17.146793 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:16:17.147170 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:17.146972 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:16:17.147170 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:17.147041 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:16:17.166672 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:17.166643 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:16:17.167224 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:17.167202 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:16:17.176915 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:17.176865 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" podStartSLOduration=7.706934642 podStartE2EDuration="27.176844534s" totalImagesPullingTime="19.469909892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:17.174995266 +0000 UTC m=+27.760797478" watchObservedRunningTime="2026-09-16 01:16:17.176844534 +0000 UTC m=+27.762646746" Sep 16 01:16:17.933269 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:17.933097 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:17.933464 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:17.933374 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:18.152996 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:18.152960 2567 generic.go:356] "Generic (PLEG): container finished" podID="3ece76da-8177-4048-97cc-df65309482a6" containerID="7753e649291e867e3fff2f093346f48ec34beaad3a50fe87d00848cb5e676c05" exitCode=0 Sep 16 01:16:18.153357 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:18.153014 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78x9p" event={"ID":"3ece76da-8177-4048-97cc-df65309482a6","Type":"ContainerDied","Data":"7753e649291e867e3fff2f093346f48ec34beaad3a50fe87d00848cb5e676c05"} Sep 16 01:16:18.916574 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:18.916543 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-btbxj"] Sep 16 01:16:18.916746 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:18.916677 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:18.916823 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:18.916778 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:18.920198 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:18.920151 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-snt2b"] Sep 16 01:16:18.920379 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:18.920302 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:18.920470 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:18.920408 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:18.920536 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:18.920498 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xgkwn"] Sep 16 01:16:18.920623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:18.920574 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:18.920703 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:18.920641 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:20.160015 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:20.159979 2567 generic.go:356] "Generic (PLEG): container finished" podID="3ece76da-8177-4048-97cc-df65309482a6" containerID="1aa3b8456a1b9dcb8970d2eac5a05c5b75da550440f73d4895a73f029f7462c9" exitCode=0 Sep 16 01:16:20.160816 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:20.160029 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78x9p" event={"ID":"3ece76da-8177-4048-97cc-df65309482a6","Type":"ContainerDied","Data":"1aa3b8456a1b9dcb8970d2eac5a05c5b75da550440f73d4895a73f029f7462c9"} Sep 16 01:16:20.933368 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:20.933188 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:20.933449 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:20.933201 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:20.933496 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:20.933459 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:20.933543 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:20.933216 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:20.933594 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:20.933510 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:20.933630 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:20.933609 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:21.164647 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:21.164615 2567 generic.go:356] "Generic (PLEG): container finished" podID="3ece76da-8177-4048-97cc-df65309482a6" containerID="cd1f6faa5b124cab1475c893575b8e9adb01b5355c99258df26471a6c1ed5ca3" exitCode=0 Sep 16 01:16:21.164973 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:21.164654 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78x9p" event={"ID":"3ece76da-8177-4048-97cc-df65309482a6","Type":"ContainerDied","Data":"cd1f6faa5b124cab1475c893575b8e9adb01b5355c99258df26471a6c1ed5ca3"} Sep 16 01:16:22.575844 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:22.575804 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:22.576265 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:22.575988 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:22.576265 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:22.576068 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret podName:ea88e142-7158-477d-b67a-1f1f450cbc6a nodeName:}" failed. No retries permitted until 2026-09-16 01:16:54.576049208 +0000 UTC m=+65.161851407 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret") pod "global-pull-secret-syncer-btbxj" (UID: "ea88e142-7158-477d-b67a-1f1f450cbc6a") : object "kube-system"/"original-pull-secret" not registered Sep 16 01:16:22.676712 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:22.676677 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:22.676891 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:22.676837 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:22.676947 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:22.676908 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs podName:00728728-c5dc-4f6c-b96b-8356103fafa5 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:54.676890633 +0000 UTC m=+65.262692830 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs") pod "network-metrics-daemon-snt2b" (UID: "00728728-c5dc-4f6c-b96b-8356103fafa5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 01:16:22.778053 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:22.778017 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf9f\" (UniqueName: \"kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f\") pod \"network-check-target-xgkwn\" (UID: \"a8146604-b0d8-41a2-bd14-3daff2e45882\") " pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:22.778194 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:22.778139 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 01:16:22.778194 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:22.778155 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 01:16:22.778194 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:22.778172 2567 projected.go:196] Error preparing data for projected volume kube-api-access-7vf9f for pod openshift-network-diagnostics/network-check-target-xgkwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:22.778321 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:22.778222 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f podName:a8146604-b0d8-41a2-bd14-3daff2e45882 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:54.778209239 +0000 UTC m=+65.364011442 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7vf9f" (UniqueName: "kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f") pod "network-check-target-xgkwn" (UID: "a8146604-b0d8-41a2-bd14-3daff2e45882") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 01:16:22.933875 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:22.933794 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:22.934021 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:22.933898 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:22.934021 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:22.933914 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:22.934021 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:22.933907 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xgkwn" podUID="a8146604-b0d8-41a2-bd14-3daff2e45882" Sep 16 01:16:22.934021 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:22.934009 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-btbxj" podUID="ea88e142-7158-477d-b67a-1f1f450cbc6a" Sep 16 01:16:22.934179 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:22.934090 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-snt2b" podUID="00728728-c5dc-4f6c-b96b-8356103fafa5" Sep 16 01:16:23.741654 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.741621 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-56.ec2.internal" event="NodeReady" Sep 16 01:16:23.742030 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.741742 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 01:16:23.772141 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.772095 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d8cdf7994-q99wn"] Sep 16 01:16:23.775285 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.775256 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.777830 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.777802 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 01:16:23.779746 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.779071 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 01:16:23.779746 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.779152 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 01:16:23.779746 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.779242 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-79qw7\"" Sep 16 01:16:23.785118 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.784113 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 01:16:23.785118 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.784705 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6zdx9"] Sep 16 01:16:23.787692 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.787671 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.788250 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.788224 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8cdf7994-q99wn"] Sep 16 01:16:23.789087 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.789067 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vsrx7"] Sep 16 01:16:23.790265 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.790246 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sq7hb\"" Sep 16 01:16:23.790379 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.790245 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 01:16:23.790379 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.790365 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 01:16:23.792190 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.792169 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vsrx7" Sep 16 01:16:23.794363 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.794346 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 01:16:23.794528 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.794511 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 01:16:23.794801 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.794518 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tgqvm\"" Sep 16 01:16:23.794801 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.794518 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 01:16:23.800455 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.800434 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6zdx9"] Sep 16 01:16:23.800923 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.800906 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vsrx7"] Sep 16 01:16:23.885775 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.885745 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6lhv\" (UniqueName: \"kubernetes.io/projected/2f80a113-872c-49ae-b785-75a115bef17a-kube-api-access-g6lhv\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.885775 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.885779 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-installation-pull-secrets\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.886023 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.885801 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdqvf\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-kube-api-access-vdqvf\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.886023 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.885821 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-image-registry-private-configuration\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.886023 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.885850 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f80a113-872c-49ae-b785-75a115bef17a-tmp-dir\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.886023 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.885910 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.886023 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.886003 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.886260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.886027 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-trusted-ca\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.886260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.886056 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-bound-sa-token\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.886260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.886080 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9a324b1-00bf-4a0d-9e52-bef0adfbcd32-cert\") pod \"ingress-canary-vsrx7\" (UID: \"b9a324b1-00bf-4a0d-9e52-bef0adfbcd32\") " pod="openshift-ingress-canary/ingress-canary-vsrx7" Sep 16 01:16:23.886260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.886104 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-ca-trust-extracted\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.886260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.886133 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f80a113-872c-49ae-b785-75a115bef17a-config-volume\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.886260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.886186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9slz\" (UniqueName: \"kubernetes.io/projected/b9a324b1-00bf-4a0d-9e52-bef0adfbcd32-kube-api-access-b9slz\") pod \"ingress-canary-vsrx7\" (UID: \"b9a324b1-00bf-4a0d-9e52-bef0adfbcd32\") " pod="openshift-ingress-canary/ingress-canary-vsrx7" Sep 16 01:16:23.886260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.886221 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-certificates\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.917920 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.917849 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:23.987065 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987022 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.987253 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987076 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-trusted-ca\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.987253 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987097 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-bound-sa-token\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.987253 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987112 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9a324b1-00bf-4a0d-9e52-bef0adfbcd32-cert\") pod \"ingress-canary-vsrx7\" (UID: \"b9a324b1-00bf-4a0d-9e52-bef0adfbcd32\") " pod="openshift-ingress-canary/ingress-canary-vsrx7" Sep 16 01:16:23.987253 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987130 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-ca-trust-extracted\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.987253 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987155 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f80a113-872c-49ae-b785-75a115bef17a-config-volume\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.987253 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:23.987174 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:23.987253 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:23.987247 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls podName:2f80a113-872c-49ae-b785-75a115bef17a nodeName:}" failed. No retries permitted until 2026-09-16 01:16:24.487227563 +0000 UTC m=+35.073029770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls") pod "dns-default-6zdx9" (UID: "2f80a113-872c-49ae-b785-75a115bef17a") : secret "dns-default-metrics-tls" not found Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987178 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9slz\" (UniqueName: \"kubernetes.io/projected/b9a324b1-00bf-4a0d-9e52-bef0adfbcd32-kube-api-access-b9slz\") pod \"ingress-canary-vsrx7\" (UID: \"b9a324b1-00bf-4a0d-9e52-bef0adfbcd32\") " pod="openshift-ingress-canary/ingress-canary-vsrx7" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987381 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-certificates\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987429 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6lhv\" (UniqueName: \"kubernetes.io/projected/2f80a113-872c-49ae-b785-75a115bef17a-kube-api-access-g6lhv\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987454 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-installation-pull-secrets\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987488 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqvf\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-kube-api-access-vdqvf\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987519 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-image-registry-private-configuration\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987555 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f80a113-872c-49ae-b785-75a115bef17a-tmp-dir\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987586 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:23.987731 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:23.987745 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8cdf7994-q99wn: secret "image-registry-tls" not found Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.987744 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-ca-trust-extracted\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:23.987803 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls podName:3af9fa53-b2cb-4c58-93c3-c790ed0467f6 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:24.487786297 +0000 UTC m=+35.073588502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls") pod "image-registry-5d8cdf7994-q99wn" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6") : secret "image-registry-tls" not found Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.988088 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f80a113-872c-49ae-b785-75a115bef17a-config-volume\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.988215 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f80a113-872c-49ae-b785-75a115bef17a-tmp-dir\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.988331 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-certificates\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.988655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.988370 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-trusted-ca\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.992623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.992574 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-installation-pull-secrets\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.992623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.992591 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-image-registry-private-configuration\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.992623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.992580 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b9a324b1-00bf-4a0d-9e52-bef0adfbcd32-cert\") pod \"ingress-canary-vsrx7\" (UID: \"b9a324b1-00bf-4a0d-9e52-bef0adfbcd32\") " pod="openshift-ingress-canary/ingress-canary-vsrx7" Sep 16 01:16:23.996476 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.996451 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-bound-sa-token\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.996596 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.996534 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6lhv\" (UniqueName: \"kubernetes.io/projected/2f80a113-872c-49ae-b785-75a115bef17a-kube-api-access-g6lhv\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:23.996831 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.996809 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdqvf\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-kube-api-access-vdqvf\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:23.996976 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:23.996956 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9slz\" (UniqueName: \"kubernetes.io/projected/b9a324b1-00bf-4a0d-9e52-bef0adfbcd32-kube-api-access-b9slz\") pod \"ingress-canary-vsrx7\" (UID: \"b9a324b1-00bf-4a0d-9e52-bef0adfbcd32\") " pod="openshift-ingress-canary/ingress-canary-vsrx7" Sep 16 01:16:24.001760 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.001735 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:24.107089 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.107047 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vsrx7" Sep 16 01:16:24.490992 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.490948 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:24.491182 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.491051 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:24.491182 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:24.491126 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:24.491182 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:24.491150 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8cdf7994-q99wn: secret "image-registry-tls" not found Sep 16 01:16:24.491379 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:24.491207 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls podName:3af9fa53-b2cb-4c58-93c3-c790ed0467f6 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:25.491188707 +0000 UTC m=+36.076990915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls") pod "image-registry-5d8cdf7994-q99wn" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6") : secret "image-registry-tls" not found Sep 16 01:16:24.491379 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:24.491229 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:24.491379 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:24.491319 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls podName:2f80a113-872c-49ae-b785-75a115bef17a nodeName:}" failed. No retries permitted until 2026-09-16 01:16:25.491300773 +0000 UTC m=+36.077102963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls") pod "dns-default-6zdx9" (UID: "2f80a113-872c-49ae-b785-75a115bef17a") : secret "dns-default-metrics-tls" not found Sep 16 01:16:24.933130 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.933053 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:24.933130 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.933069 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:24.933687 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.933239 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:24.936656 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.936632 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 01:16:24.936656 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.936647 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zwsmz\"" Sep 16 01:16:24.936836 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.936663 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 01:16:24.936836 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.936682 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 01:16:24.936836 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.936689 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 01:16:24.936972 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:24.936897 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6h4t\"" Sep 16 01:16:25.497681 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:25.497649 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:25.497832 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:25.497709 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:25.497832 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:25.497804 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:25.497832 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:25.497816 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:25.497832 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:25.497823 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8cdf7994-q99wn: secret "image-registry-tls" not found Sep 16 01:16:25.497952 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:25.497867 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls podName:2f80a113-872c-49ae-b785-75a115bef17a nodeName:}" failed. No retries permitted until 2026-09-16 01:16:27.497852408 +0000 UTC m=+38.083654609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls") pod "dns-default-6zdx9" (UID: "2f80a113-872c-49ae-b785-75a115bef17a") : secret "dns-default-metrics-tls" not found Sep 16 01:16:25.497952 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:25.497880 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls podName:3af9fa53-b2cb-4c58-93c3-c790ed0467f6 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:27.49787424 +0000 UTC m=+38.083676428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls") pod "image-registry-5d8cdf7994-q99wn" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6") : secret "image-registry-tls" not found Sep 16 01:16:25.588946 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:25.588879 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:25.638476 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:25.638379 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:25.675340 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:25.675313 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vsrx7"] Sep 16 01:16:25.679079 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:16:25.679047 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a324b1_00bf_4a0d_9e52_bef0adfbcd32.slice/crio-5025852bc3dc694d06a4eef3a463eb2a401c68ac8d26cd296bf1c671138f06b9 WatchSource:0}: Error finding container 5025852bc3dc694d06a4eef3a463eb2a401c68ac8d26cd296bf1c671138f06b9: Status 404 returned error can't find the container with id 5025852bc3dc694d06a4eef3a463eb2a401c68ac8d26cd296bf1c671138f06b9 Sep 16 01:16:26.179610 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:26.179380 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vsrx7" event={"ID":"b9a324b1-00bf-4a0d-9e52-bef0adfbcd32","Type":"ContainerStarted","Data":"5025852bc3dc694d06a4eef3a463eb2a401c68ac8d26cd296bf1c671138f06b9"} Sep 16 01:16:26.182188 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:26.182159 2567 generic.go:356] "Generic (PLEG): container finished" podID="3ece76da-8177-4048-97cc-df65309482a6" containerID="700ed27ab790a72622b2734e940f7543905f9afbd2f79d9fefad3401938615cb" exitCode=0 Sep 16 01:16:26.182319 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:26.182210 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78x9p" event={"ID":"3ece76da-8177-4048-97cc-df65309482a6","Type":"ContainerDied","Data":"700ed27ab790a72622b2734e940f7543905f9afbd2f79d9fefad3401938615cb"} Sep 16 01:16:27.187403 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:27.187369 2567 generic.go:356] "Generic (PLEG): container finished" podID="3ece76da-8177-4048-97cc-df65309482a6" containerID="10abe8540f840dcb9251d15c6f75317eac9a7529d10b3f3b841f0672593c0fb3" exitCode=0 Sep 16 01:16:27.187855 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:27.187422 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78x9p" event={"ID":"3ece76da-8177-4048-97cc-df65309482a6","Type":"ContainerDied","Data":"10abe8540f840dcb9251d15c6f75317eac9a7529d10b3f3b841f0672593c0fb3"} Sep 16 01:16:27.516632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:27.516596 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:27.516801 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:27.516682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:27.516801 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:27.516761 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:27.516801 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:27.516779 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8cdf7994-q99wn: secret "image-registry-tls" not found Sep 16 01:16:27.516801 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:27.516779 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:27.517025 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:27.516842 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls podName:2f80a113-872c-49ae-b785-75a115bef17a nodeName:}" failed. No retries permitted until 2026-09-16 01:16:31.516824674 +0000 UTC m=+42.102626863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls") pod "dns-default-6zdx9" (UID: "2f80a113-872c-49ae-b785-75a115bef17a") : secret "dns-default-metrics-tls" not found Sep 16 01:16:27.517025 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:27.516856 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls podName:3af9fa53-b2cb-4c58-93c3-c790ed0467f6 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:31.516850418 +0000 UTC m=+42.102652606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls") pod "image-registry-5d8cdf7994-q99wn" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6") : secret "image-registry-tls" not found Sep 16 01:16:28.192409 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:28.192333 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-78x9p" event={"ID":"3ece76da-8177-4048-97cc-df65309482a6","Type":"ContainerStarted","Data":"a9a7d231e64ff72f4a74aaa81be5b45697f4274e23c619ec535b7963e4de4352"} Sep 16 01:16:28.213018 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:28.212969 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-78x9p" podStartSLOduration=7.250877544 podStartE2EDuration="38.212951985s" totalImagesPullingTime="30.962074441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:28.211488898 +0000 UTC m=+38.797291135" watchObservedRunningTime="2026-09-16 01:16:28.212951985 +0000 UTC m=+38.798754186" Sep 16 01:16:28.330364 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:28.330326 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:28.382435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:28.382407 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:28.436972 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:28.436937 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:28.538036 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:28.537996 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:28.638136 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:28.638087 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:28.737656 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:28.737629 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:28.838408 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:28.838366 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:29.196598 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:29.196505 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vsrx7" event={"ID":"b9a324b1-00bf-4a0d-9e52-bef0adfbcd32","Type":"ContainerStarted","Data":"d60c3f50cefd662d853bae247f91119dcf95f08937571a07bcdd97421ecf3a26"} Sep 16 01:16:29.213336 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:29.213266 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vsrx7" podStartSLOduration=3.248326874 podStartE2EDuration="6.213249561s" totalImagesPullingTime="2.964922687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:16:29.211942464 +0000 UTC m=+39.797744675" watchObservedRunningTime="2026-09-16 01:16:29.213249561 +0000 UTC m=+39.799051772" Sep 16 01:16:30.968818 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:30.968784 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:31.015604 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:31.015561 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" limit=1024 Sep 16 01:16:31.546611 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:31.546572 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:31.546765 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:31.546635 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:31.546765 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:31.546716 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:31.546859 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:31.546777 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls podName:2f80a113-872c-49ae-b785-75a115bef17a nodeName:}" failed. No retries permitted until 2026-09-16 01:16:39.546762417 +0000 UTC m=+50.132564606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls") pod "dns-default-6zdx9" (UID: "2f80a113-872c-49ae-b785-75a115bef17a") : secret "dns-default-metrics-tls" not found Sep 16 01:16:31.546859 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:31.546718 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:31.546859 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:31.546820 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8cdf7994-q99wn: secret "image-registry-tls" not found Sep 16 01:16:31.546859 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:31.546856 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls podName:3af9fa53-b2cb-4c58-93c3-c790ed0467f6 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:39.546845385 +0000 UTC m=+50.132647598 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls") pod "image-registry-5d8cdf7994-q99wn" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6") : secret "image-registry-tls" not found Sep 16 01:16:39.609739 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:39.609702 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:39.610208 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:39.609772 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:39.610208 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:39.609874 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:39.610208 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:39.609893 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8cdf7994-q99wn: secret "image-registry-tls" not found Sep 16 01:16:39.610208 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:39.609907 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:39.610208 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:39.609963 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls podName:3af9fa53-b2cb-4c58-93c3-c790ed0467f6 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:55.609945507 +0000 UTC m=+66.195747709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls") pod "image-registry-5d8cdf7994-q99wn" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6") : secret "image-registry-tls" not found Sep 16 01:16:39.610208 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:39.609981 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls podName:2f80a113-872c-49ae-b785-75a115bef17a nodeName:}" failed. No retries permitted until 2026-09-16 01:16:55.609973184 +0000 UTC m=+66.195775373 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls") pod "dns-default-6zdx9" (UID: "2f80a113-872c-49ae-b785-75a115bef17a") : secret "dns-default-metrics-tls" not found Sep 16 01:16:49.178266 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.178238 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-79tjp" Sep 16 01:16:49.615819 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.615783 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j"] Sep 16 01:16:49.618489 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.618471 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:16:49.623650 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.623622 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dwkcc\"" Sep 16 01:16:49.623984 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.623967 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 16 01:16:49.624076 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.623985 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 16 01:16:49.624201 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.624187 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 16 01:16:49.629240 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.629220 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j"] Sep 16 01:16:49.715474 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.715444 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd"] Sep 16 01:16:49.717226 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.717211 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:49.719654 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.719628 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 01:16:49.719654 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.719628 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 01:16:49.719846 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.719660 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 16 01:16:49.719957 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.719941 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-h8fbk\"" Sep 16 01:16:49.721148 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.721133 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 16 01:16:49.727104 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.727081 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd"] Sep 16 01:16:49.789433 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.789395 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xp44j\" (UID: \"32850513-a013-4d35-8e7e-0946fef7b608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:16:49.789597 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.789454 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsffb\" (UniqueName: \"kubernetes.io/projected/32850513-a013-4d35-8e7e-0946fef7b608-kube-api-access-fsffb\") pod \"cluster-samples-operator-86499c89c4-xp44j\" (UID: \"32850513-a013-4d35-8e7e-0946fef7b608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:16:49.889882 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.889796 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:49.889882 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.889843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xp44j\" (UID: \"32850513-a013-4d35-8e7e-0946fef7b608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:16:49.889882 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.889873 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz5d8\" (UniqueName: \"kubernetes.io/projected/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-kube-api-access-kz5d8\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:49.890125 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.889927 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsffb\" (UniqueName: \"kubernetes.io/projected/32850513-a013-4d35-8e7e-0946fef7b608-kube-api-access-fsffb\") pod \"cluster-samples-operator-86499c89c4-xp44j\" (UID: \"32850513-a013-4d35-8e7e-0946fef7b608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:16:49.890125 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.890024 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:49.892193 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.892173 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 16 01:16:49.897230 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.897213 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 16 01:16:49.900005 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:49.899989 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 01:16:49.900093 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:49.900055 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls podName:32850513-a013-4d35-8e7e-0946fef7b608 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:50.400034368 +0000 UTC m=+60.985836562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xp44j" (UID: "32850513-a013-4d35-8e7e-0946fef7b608") : secret "samples-operator-tls" not found Sep 16 01:16:49.908255 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.908238 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 16 01:16:49.920772 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.920750 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsffb\" (UniqueName: \"kubernetes.io/projected/32850513-a013-4d35-8e7e-0946fef7b608-kube-api-access-fsffb\") pod \"cluster-samples-operator-86499c89c4-xp44j\" (UID: \"32850513-a013-4d35-8e7e-0946fef7b608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:16:49.991143 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.991093 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:49.991143 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.991144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:49.991373 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.991236 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz5d8\" (UniqueName: \"kubernetes.io/projected/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-kube-api-access-kz5d8\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:49.993636 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.993620 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 16 01:16:49.995680 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.995653 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 16 01:16:49.999245 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:49.999227 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 01:16:50.002166 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:50.002151 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 01:16:50.002226 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:50.002215 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls podName:9ae8823d-2dd3-4b66-a92a-aaca58fad1c6 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:50.502192349 +0000 UTC m=+61.087994540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zs9pd" (UID: "9ae8823d-2dd3-4b66-a92a-aaca58fad1c6") : secret "cluster-monitoring-operator-tls" not found Sep 16 01:16:50.002710 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:50.002691 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:50.009653 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:50.009635 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 01:16:50.021265 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:50.021238 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz5d8\" (UniqueName: \"kubernetes.io/projected/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-kube-api-access-kz5d8\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:50.494835 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:50.494777 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xp44j\" (UID: \"32850513-a013-4d35-8e7e-0946fef7b608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:16:50.495369 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:50.494925 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 01:16:50.495369 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:50.494984 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls podName:32850513-a013-4d35-8e7e-0946fef7b608 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:51.494970373 +0000 UTC m=+62.080772562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xp44j" (UID: "32850513-a013-4d35-8e7e-0946fef7b608") : secret "samples-operator-tls" not found Sep 16 01:16:50.595618 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:50.595582 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:50.595752 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:50.595727 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 01:16:50.595801 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:50.595791 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls podName:9ae8823d-2dd3-4b66-a92a-aaca58fad1c6 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:51.595772897 +0000 UTC m=+62.181575101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zs9pd" (UID: "9ae8823d-2dd3-4b66-a92a-aaca58fad1c6") : secret "cluster-monitoring-operator-tls" not found Sep 16 01:16:51.501739 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:51.501704 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xp44j\" (UID: \"32850513-a013-4d35-8e7e-0946fef7b608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:16:51.502119 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:51.501822 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 01:16:51.502119 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:51.501877 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls podName:32850513-a013-4d35-8e7e-0946fef7b608 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:53.501863196 +0000 UTC m=+64.087665384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xp44j" (UID: "32850513-a013-4d35-8e7e-0946fef7b608") : secret "samples-operator-tls" not found Sep 16 01:16:51.602046 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:51.602005 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:51.602198 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:51.602141 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 01:16:51.602242 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:51.602200 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls podName:9ae8823d-2dd3-4b66-a92a-aaca58fad1c6 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:53.60218207 +0000 UTC m=+64.187984262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zs9pd" (UID: "9ae8823d-2dd3-4b66-a92a-aaca58fad1c6") : secret "cluster-monitoring-operator-tls" not found Sep 16 01:16:53.515138 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:53.515091 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xp44j\" (UID: \"32850513-a013-4d35-8e7e-0946fef7b608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:16:53.515531 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:53.515236 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 01:16:53.515531 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:53.515327 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls podName:32850513-a013-4d35-8e7e-0946fef7b608 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:57.515308903 +0000 UTC m=+68.101111093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xp44j" (UID: "32850513-a013-4d35-8e7e-0946fef7b608") : secret "samples-operator-tls" not found Sep 16 01:16:53.616185 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:53.616152 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:53.616359 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:53.616271 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 01:16:53.616359 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:53.616364 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls podName:9ae8823d-2dd3-4b66-a92a-aaca58fad1c6 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:57.616348736 +0000 UTC m=+68.202150938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zs9pd" (UID: "9ae8823d-2dd3-4b66-a92a-aaca58fad1c6") : secret "cluster-monitoring-operator-tls" not found Sep 16 01:16:54.235839 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.235805 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kmhgw"] Sep 16 01:16:54.238421 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.238397 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.241100 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.241071 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 01:16:54.241215 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.241101 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ncdjs\"" Sep 16 01:16:54.241215 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.241174 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 01:16:54.242416 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.242398 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 01:16:54.242416 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.242410 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 01:16:54.248843 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.248823 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kmhgw"] Sep 16 01:16:54.320809 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.320772 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/799379d2-81b4-458b-9a29-5c43f37c50f1-data-volume\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.321006 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.320820 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zggcp\" (UniqueName: \"kubernetes.io/projected/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-api-access-zggcp\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.321006 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.320861 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.321006 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.320924 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/799379d2-81b4-458b-9a29-5c43f37c50f1-crio-socket\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.321006 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.320946 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.422232 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.422196 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/799379d2-81b4-458b-9a29-5c43f37c50f1-data-volume\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.422420 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.422243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zggcp\" (UniqueName: \"kubernetes.io/projected/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-api-access-zggcp\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.422420 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.422301 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.422420 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.422331 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/799379d2-81b4-458b-9a29-5c43f37c50f1-crio-socket\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.422420 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.422354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.422645 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:54.422455 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:16:54.422645 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.422456 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/799379d2-81b4-458b-9a29-5c43f37c50f1-crio-socket\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.422645 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:54.422459 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 01:16:54.422645 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:54.422574 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls podName:799379d2-81b4-458b-9a29-5c43f37c50f1 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:54.922554545 +0000 UTC m=+65.508356737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmhgw" (UID: "799379d2-81b4-458b-9a29-5c43f37c50f1") : secret "insights-runtime-extractor-tls" not found Sep 16 01:16:54.422645 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.422623 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/799379d2-81b4-458b-9a29-5c43f37c50f1-data-volume\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.422645 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:54.422628 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm podName:799379d2-81b4-458b-9a29-5c43f37c50f1 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:54.92261144 +0000 UTC m=+65.508413646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmhgw" (UID: "799379d2-81b4-458b-9a29-5c43f37c50f1") : configmap "kube-rbac-proxy" not found Sep 16 01:16:54.430066 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.430041 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zggcp\" (UniqueName: \"kubernetes.io/projected/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-api-access-zggcp\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.623709 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.623627 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:54.626233 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.626214 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 01:16:54.636632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.636608 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ea88e142-7158-477d-b67a-1f1f450cbc6a-original-pull-secret\") pod \"global-pull-secret-syncer-btbxj\" (UID: \"ea88e142-7158-477d-b67a-1f1f450cbc6a\") " pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:54.651735 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.651703 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-btbxj" Sep 16 01:16:54.724045 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.724012 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:16:54.726473 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.726447 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 01:16:54.734857 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:54.734835 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 01:16:54.734947 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:54.734900 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs podName:00728728-c5dc-4f6c-b96b-8356103fafa5 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:58.734883225 +0000 UTC m=+129.320685428 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs") pod "network-metrics-daemon-snt2b" (UID: "00728728-c5dc-4f6c-b96b-8356103fafa5") : secret "metrics-daemon-secret" not found Sep 16 01:16:54.824457 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.824428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vf9f\" (UniqueName: \"kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f\") pod \"network-check-target-xgkwn\" (UID: \"a8146604-b0d8-41a2-bd14-3daff2e45882\") " pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:54.826883 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.826863 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 01:16:54.837506 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.837478 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 01:16:54.849122 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.849098 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vf9f\" (UniqueName: \"kubernetes.io/projected/a8146604-b0d8-41a2-bd14-3daff2e45882-kube-api-access-7vf9f\") pod \"network-check-target-xgkwn\" (UID: \"a8146604-b0d8-41a2-bd14-3daff2e45882\") " pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:54.917690 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.917619 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-btbxj"] Sep 16 01:16:54.921056 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:16:54.921035 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea88e142_7158_477d_b67a_1f1f450cbc6a.slice/crio-a4626ad9d83890912b9c89cbe9b3a12ceb8f7fe38391c47c24428b5c4463f7a5 WatchSource:0}: Error finding container a4626ad9d83890912b9c89cbe9b3a12ceb8f7fe38391c47c24428b5c4463f7a5: Status 404 returned error can't find the container with id a4626ad9d83890912b9c89cbe9b3a12ceb8f7fe38391c47c24428b5c4463f7a5 Sep 16 01:16:54.926936 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.926908 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.927068 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.927049 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:54.927068 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:54.927059 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 01:16:54.927179 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:54.927124 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm podName:799379d2-81b4-458b-9a29-5c43f37c50f1 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:55.927103267 +0000 UTC m=+66.512905458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmhgw" (UID: "799379d2-81b4-458b-9a29-5c43f37c50f1") : configmap "kube-rbac-proxy" not found Sep 16 01:16:54.927179 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:54.927164 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:16:54.927248 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:54.927214 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls podName:799379d2-81b4-458b-9a29-5c43f37c50f1 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:55.927201332 +0000 UTC m=+66.513003521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmhgw" (UID: "799379d2-81b4-458b-9a29-5c43f37c50f1") : secret "insights-runtime-extractor-tls" not found Sep 16 01:16:54.959380 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.959351 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zwsmz\"" Sep 16 01:16:54.967217 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:54.967189 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:16:55.236847 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:55.236821 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xgkwn"] Sep 16 01:16:55.239783 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:16:55.239750 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8146604_b0d8_41a2_bd14_3daff2e45882.slice/crio-97822e1920d97977d12a9baac686c669dcfe3b9fa18b0cc37deed9607708c5f3 WatchSource:0}: Error finding container 97822e1920d97977d12a9baac686c669dcfe3b9fa18b0cc37deed9607708c5f3: Status 404 returned error can't find the container with id 97822e1920d97977d12a9baac686c669dcfe3b9fa18b0cc37deed9607708c5f3 Sep 16 01:16:55.258496 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:55.258469 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xgkwn" event={"ID":"a8146604-b0d8-41a2-bd14-3daff2e45882","Type":"ContainerStarted","Data":"97822e1920d97977d12a9baac686c669dcfe3b9fa18b0cc37deed9607708c5f3"} Sep 16 01:16:55.259548 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:55.259521 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-btbxj" event={"ID":"ea88e142-7158-477d-b67a-1f1f450cbc6a","Type":"ContainerStarted","Data":"a4626ad9d83890912b9c89cbe9b3a12ceb8f7fe38391c47c24428b5c4463f7a5"} Sep 16 01:16:55.631648 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:55.631568 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:16:55.631648 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:55.631615 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:16:55.632016 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:55.631708 2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 01:16:55.632016 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:55.631720 2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 01:16:55.632016 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:55.631741 2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d8cdf7994-q99wn: secret "image-registry-tls" not found Sep 16 01:16:55.632016 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:55.631761 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls podName:2f80a113-872c-49ae-b785-75a115bef17a nodeName:}" failed. No retries permitted until 2026-09-16 01:17:27.631747898 +0000 UTC m=+98.217550088 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls") pod "dns-default-6zdx9" (UID: "2f80a113-872c-49ae-b785-75a115bef17a") : secret "dns-default-metrics-tls" not found Sep 16 01:16:55.632016 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:55.631789 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls podName:3af9fa53-b2cb-4c58-93c3-c790ed0467f6 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:27.63177707 +0000 UTC m=+98.217579259 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls") pod "image-registry-5d8cdf7994-q99wn" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6") : secret "image-registry-tls" not found Sep 16 01:16:55.934388 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:55.934297 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:55.934388 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:55.934354 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:55.934608 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:55.934455 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 01:16:55.934608 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:55.934466 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:16:55.934608 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:55.934509 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm podName:799379d2-81b4-458b-9a29-5c43f37c50f1 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:57.934495841 +0000 UTC m=+68.520298030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmhgw" (UID: "799379d2-81b4-458b-9a29-5c43f37c50f1") : configmap "kube-rbac-proxy" not found Sep 16 01:16:55.934608 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:55.934533 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls podName:799379d2-81b4-458b-9a29-5c43f37c50f1 nodeName:}" failed. No retries permitted until 2026-09-16 01:16:57.934515711 +0000 UTC m=+68.520317924 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmhgw" (UID: "799379d2-81b4-458b-9a29-5c43f37c50f1") : secret "insights-runtime-extractor-tls" not found Sep 16 01:16:55.984365 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:55.984335 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9glj9_b01ecda8-8841-4584-a12b-daff7da47a47/dns-node-resolver/0.log" Sep 16 01:16:56.906383 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:56.906355 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-cqhvn"] Sep 16 01:16:56.908476 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:56.908453 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" Sep 16 01:16:56.911328 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:56.910830 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-48fft\"" Sep 16 01:16:56.911328 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:56.910947 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 16 01:16:56.912030 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:56.912007 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 16 01:16:56.912222 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:56.912206 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 16 01:16:56.912462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:56.912448 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 16 01:16:56.915968 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:56.915943 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-cqhvn"] Sep 16 01:16:56.942667 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:56.942640 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/92c12008-9fd5-4f1d-a601-febad35abcb0-signing-key\") pod \"service-ca-568869fdc5-cqhvn\" (UID: \"92c12008-9fd5-4f1d-a601-febad35abcb0\") " pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" Sep 16 01:16:56.942818 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:56.942688 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktlrq\" (UniqueName: \"kubernetes.io/projected/92c12008-9fd5-4f1d-a601-febad35abcb0-kube-api-access-ktlrq\") pod \"service-ca-568869fdc5-cqhvn\" (UID: \"92c12008-9fd5-4f1d-a601-febad35abcb0\") " pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" Sep 16 01:16:56.942881 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:56.942808 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/92c12008-9fd5-4f1d-a601-febad35abcb0-signing-cabundle\") pod \"service-ca-568869fdc5-cqhvn\" (UID: \"92c12008-9fd5-4f1d-a601-febad35abcb0\") " pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" Sep 16 01:16:57.044324 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.044216 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/92c12008-9fd5-4f1d-a601-febad35abcb0-signing-key\") pod \"service-ca-568869fdc5-cqhvn\" (UID: \"92c12008-9fd5-4f1d-a601-febad35abcb0\") " pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" Sep 16 01:16:57.044324 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.044325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktlrq\" (UniqueName: \"kubernetes.io/projected/92c12008-9fd5-4f1d-a601-febad35abcb0-kube-api-access-ktlrq\") pod \"service-ca-568869fdc5-cqhvn\" (UID: \"92c12008-9fd5-4f1d-a601-febad35abcb0\") " pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" Sep 16 01:16:57.044573 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.044392 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/92c12008-9fd5-4f1d-a601-febad35abcb0-signing-cabundle\") pod \"service-ca-568869fdc5-cqhvn\" (UID: \"92c12008-9fd5-4f1d-a601-febad35abcb0\") " pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" Sep 16 01:16:57.045084 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.045058 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/92c12008-9fd5-4f1d-a601-febad35abcb0-signing-cabundle\") pod \"service-ca-568869fdc5-cqhvn\" (UID: \"92c12008-9fd5-4f1d-a601-febad35abcb0\") " pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" Sep 16 01:16:57.047525 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.047484 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/92c12008-9fd5-4f1d-a601-febad35abcb0-signing-key\") pod \"service-ca-568869fdc5-cqhvn\" (UID: \"92c12008-9fd5-4f1d-a601-febad35abcb0\") " pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" Sep 16 01:16:57.053167 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.053140 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktlrq\" (UniqueName: \"kubernetes.io/projected/92c12008-9fd5-4f1d-a601-febad35abcb0-kube-api-access-ktlrq\") pod \"service-ca-568869fdc5-cqhvn\" (UID: \"92c12008-9fd5-4f1d-a601-febad35abcb0\") " pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" Sep 16 01:16:57.220257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.220183 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" Sep 16 01:16:57.385007 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.384981 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sxdzj_dc623d5a-e0b2-4aa5-967f-c8b417121518/node-ca/0.log" Sep 16 01:16:57.502040 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.502003 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-cqhvn"] Sep 16 01:16:57.505878 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:16:57.505843 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c12008_9fd5_4f1d_a601_febad35abcb0.slice/crio-7d8a82a9cf680f4f6966362d757f93b51566e872db59c80ce7d85490f16174fa WatchSource:0}: Error finding container 7d8a82a9cf680f4f6966362d757f93b51566e872db59c80ce7d85490f16174fa: Status 404 returned error can't find the container with id 7d8a82a9cf680f4f6966362d757f93b51566e872db59c80ce7d85490f16174fa Sep 16 01:16:57.549405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.549366 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xp44j\" (UID: \"32850513-a013-4d35-8e7e-0946fef7b608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:16:57.549558 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:57.549504 2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 01:16:57.549606 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:57.549564 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls podName:32850513-a013-4d35-8e7e-0946fef7b608 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:05.549550807 +0000 UTC m=+76.135352995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-xp44j" (UID: "32850513-a013-4d35-8e7e-0946fef7b608") : secret "samples-operator-tls" not found Sep 16 01:16:57.650129 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.650086 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:16:57.650346 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:57.650225 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 01:16:57.650346 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:57.650330 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls podName:9ae8823d-2dd3-4b66-a92a-aaca58fad1c6 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:05.65030879 +0000 UTC m=+76.236110993 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zs9pd" (UID: "9ae8823d-2dd3-4b66-a92a-aaca58fad1c6") : secret "cluster-monitoring-operator-tls" not found Sep 16 01:16:57.953423 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.953317 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:57.953423 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.953396 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:16:57.953919 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:57.953493 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:16:57.953919 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:57.953527 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 01:16:57.953919 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:57.953579 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls podName:799379d2-81b4-458b-9a29-5c43f37c50f1 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:01.953555949 +0000 UTC m=+72.539358142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmhgw" (UID: "799379d2-81b4-458b-9a29-5c43f37c50f1") : secret "insights-runtime-extractor-tls" not found Sep 16 01:16:57.953919 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:16:57.953598 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm podName:799379d2-81b4-458b-9a29-5c43f37c50f1 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:01.953589936 +0000 UTC m=+72.539392126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmhgw" (UID: "799379d2-81b4-458b-9a29-5c43f37c50f1") : configmap "kube-rbac-proxy" not found Sep 16 01:16:57.984669 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:57.984643 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vsrx7_b9a324b1-00bf-4a0d-9e52-bef0adfbcd32/serve-healthcheck-canary/0.log" Sep 16 01:16:58.269385 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:16:58.269341 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" event={"ID":"92c12008-9fd5-4f1d-a601-febad35abcb0","Type":"ContainerStarted","Data":"7d8a82a9cf680f4f6966362d757f93b51566e872db59c80ce7d85490f16174fa"} Sep 16 01:17:00.277515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:00.277476 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xgkwn" event={"ID":"a8146604-b0d8-41a2-bd14-3daff2e45882","Type":"ContainerStarted","Data":"fa0069c443da2892e767d4f5ab654f1ac8fe95ca79341e759cf651d169543e73"} Sep 16 01:17:00.277915 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:00.277591 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:17:00.279237 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:00.279202 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-btbxj" event={"ID":"ea88e142-7158-477d-b67a-1f1f450cbc6a","Type":"ContainerStarted","Data":"5717bcc89abcfc9fac8183386879816d3ccf85c6919323645e8de9554fcf42b9"} Sep 16 01:17:00.295960 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:00.295910 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xgkwn" podStartSLOduration=65.628642937 podStartE2EDuration="1m10.295895261s" totalImagesPullingTime="4.667252324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:00.294770245 +0000 UTC m=+70.880572457" watchObservedRunningTime="2026-09-16 01:17:00.295895261 +0000 UTC m=+70.881697474" Sep 16 01:17:00.308799 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:00.308739 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-btbxj" podStartSLOduration=65.318430142 podStartE2EDuration="1m10.308721293s" totalImagesPullingTime="4.990291151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:00.30824725 +0000 UTC m=+70.894049462" watchObservedRunningTime="2026-09-16 01:17:00.308721293 +0000 UTC m=+70.894523503" Sep 16 01:17:01.986515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:01.986472 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:17:01.986949 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:01.986523 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:17:01.986949 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:01.986638 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:17:01.986949 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:01.986668 2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 01:17:01.986949 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:01.986704 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls podName:799379d2-81b4-458b-9a29-5c43f37c50f1 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:09.986686087 +0000 UTC m=+80.572488458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmhgw" (UID: "799379d2-81b4-458b-9a29-5c43f37c50f1") : secret "insights-runtime-extractor-tls" not found Sep 16 01:17:01.986949 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:01.986738 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm podName:799379d2-81b4-458b-9a29-5c43f37c50f1 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:09.986723467 +0000 UTC m=+80.572525669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmhgw" (UID: "799379d2-81b4-458b-9a29-5c43f37c50f1") : configmap "kube-rbac-proxy" not found Sep 16 01:17:02.286517 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:02.286432 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" event={"ID":"92c12008-9fd5-4f1d-a601-febad35abcb0","Type":"ContainerStarted","Data":"cd53310f576c080422d3af6632357db944cfe6c295f229506509b49b144c87b4"} Sep 16 01:17:02.301044 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:02.300989 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-cqhvn" podStartSLOduration=2.436915514 podStartE2EDuration="6.300973921s" totalImagesPullingTime="3.864058407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:02.299836152 +0000 UTC m=+72.885638362" watchObservedRunningTime="2026-09-16 01:17:02.300973921 +0000 UTC m=+72.886776132" Sep 16 01:17:05.617219 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:05.617176 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xp44j\" (UID: \"32850513-a013-4d35-8e7e-0946fef7b608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:17:05.619767 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:05.619739 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/32850513-a013-4d35-8e7e-0946fef7b608-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-xp44j\" (UID: \"32850513-a013-4d35-8e7e-0946fef7b608\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:17:05.718316 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:05.718249 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:17:05.718513 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:05.718406 2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 01:17:05.718513 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:05.718487 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls podName:9ae8823d-2dd3-4b66-a92a-aaca58fad1c6 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:21.718470419 +0000 UTC m=+92.304272626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-zs9pd" (UID: "9ae8823d-2dd3-4b66-a92a-aaca58fad1c6") : secret "cluster-monitoring-operator-tls" not found Sep 16 01:17:05.829758 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:05.829727 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dwkcc\"" Sep 16 01:17:05.837779 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:05.837759 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" Sep 16 01:17:06.105144 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:06.105114 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j"] Sep 16 01:17:06.298091 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:06.298048 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" event={"ID":"32850513-a013-4d35-8e7e-0946fef7b608","Type":"ContainerStarted","Data":"3b8a1d8320029140ebaa12c15539431ece1ef3eb161721e8ca2581552b06f38b"} Sep 16 01:17:08.306327 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:08.306291 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" event={"ID":"32850513-a013-4d35-8e7e-0946fef7b608","Type":"ContainerStarted","Data":"91133fdee452bbe81da61e91e8a5bcea0e060f208a11541ffde6ceac7c61fc58"} Sep 16 01:17:08.306704 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:08.306341 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" event={"ID":"32850513-a013-4d35-8e7e-0946fef7b608","Type":"ContainerStarted","Data":"5f495e2651c8b7173179c21997c90cbbd64e7a3042083d06084410efbfcb7bae"} Sep 16 01:17:08.321451 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:08.321371 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-xp44j" podStartSLOduration=17.708008924 podStartE2EDuration="19.321311147s" totalImagesPullingTime="1.613302223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:08.320258729 +0000 UTC m=+78.906060940" watchObservedRunningTime="2026-09-16 01:17:08.321311147 +0000 UTC m=+78.907113358" Sep 16 01:17:10.054399 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:10.054346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:17:10.054836 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:10.054411 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:17:10.054836 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:10.054501 2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 01:17:10.054836 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:10.054570 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls podName:799379d2-81b4-458b-9a29-5c43f37c50f1 nodeName:}" failed. No retries permitted until 2026-09-16 01:17:26.054554111 +0000 UTC m=+96.640356300 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmhgw" (UID: "799379d2-81b4-458b-9a29-5c43f37c50f1") : secret "insights-runtime-extractor-tls" not found Sep 16 01:17:10.055083 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:10.055059 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/799379d2-81b4-458b-9a29-5c43f37c50f1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:17:18.639790 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.639754 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8"] Sep 16 01:17:18.644106 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.644086 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:18.647608 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.647583 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 01:17:18.647735 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.647615 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 16 01:17:18.647735 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.647653 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 01:17:18.647735 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.647731 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 01:17:18.652135 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.652116 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8"] Sep 16 01:17:18.720330 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.719460 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2c9cb20b-07e2-4f91-8693-189a771aaf1c-klusterlet-config\") pod \"klusterlet-addon-workmgr-54f4f89bf7-q9lz8\" (UID: \"2c9cb20b-07e2-4f91-8693-189a771aaf1c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:18.720330 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.719654 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb20b-07e2-4f91-8693-189a771aaf1c-tmp\") pod \"klusterlet-addon-workmgr-54f4f89bf7-q9lz8\" (UID: \"2c9cb20b-07e2-4f91-8693-189a771aaf1c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:18.720330 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.719697 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6hh\" (UniqueName: \"kubernetes.io/projected/2c9cb20b-07e2-4f91-8693-189a771aaf1c-kube-api-access-9d6hh\") pod \"klusterlet-addon-workmgr-54f4f89bf7-q9lz8\" (UID: \"2c9cb20b-07e2-4f91-8693-189a771aaf1c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:18.742792 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.742752 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-gkl42"] Sep 16 01:17:18.745663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.745642 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-gkl42" Sep 16 01:17:18.748621 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.748599 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9jktp\"" Sep 16 01:17:18.748731 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.748633 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 01:17:18.748731 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.748642 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 01:17:18.761823 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.761799 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-gkl42"] Sep 16 01:17:18.820641 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.820607 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb20b-07e2-4f91-8693-189a771aaf1c-tmp\") pod \"klusterlet-addon-workmgr-54f4f89bf7-q9lz8\" (UID: \"2c9cb20b-07e2-4f91-8693-189a771aaf1c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:18.820831 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.820653 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6hh\" (UniqueName: \"kubernetes.io/projected/2c9cb20b-07e2-4f91-8693-189a771aaf1c-kube-api-access-9d6hh\") pod \"klusterlet-addon-workmgr-54f4f89bf7-q9lz8\" (UID: \"2c9cb20b-07e2-4f91-8693-189a771aaf1c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:18.820831 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.820690 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9s7q\" (UniqueName: \"kubernetes.io/projected/c28672a2-2930-4f95-80b3-0cccb3ce00d9-kube-api-access-n9s7q\") pod \"downloads-76f99584f4-gkl42\" (UID: \"c28672a2-2930-4f95-80b3-0cccb3ce00d9\") " pod="openshift-console/downloads-76f99584f4-gkl42" Sep 16 01:17:18.820935 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.820826 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2c9cb20b-07e2-4f91-8693-189a771aaf1c-klusterlet-config\") pod \"klusterlet-addon-workmgr-54f4f89bf7-q9lz8\" (UID: \"2c9cb20b-07e2-4f91-8693-189a771aaf1c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:18.821012 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.820993 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb20b-07e2-4f91-8693-189a771aaf1c-tmp\") pod \"klusterlet-addon-workmgr-54f4f89bf7-q9lz8\" (UID: \"2c9cb20b-07e2-4f91-8693-189a771aaf1c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:18.823404 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.823381 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2c9cb20b-07e2-4f91-8693-189a771aaf1c-klusterlet-config\") pod \"klusterlet-addon-workmgr-54f4f89bf7-q9lz8\" (UID: \"2c9cb20b-07e2-4f91-8693-189a771aaf1c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:18.829657 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.829634 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6hh\" (UniqueName: \"kubernetes.io/projected/2c9cb20b-07e2-4f91-8693-189a771aaf1c-kube-api-access-9d6hh\") pod \"klusterlet-addon-workmgr-54f4f89bf7-q9lz8\" (UID: \"2c9cb20b-07e2-4f91-8693-189a771aaf1c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:18.921402 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.921314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9s7q\" (UniqueName: \"kubernetes.io/projected/c28672a2-2930-4f95-80b3-0cccb3ce00d9-kube-api-access-n9s7q\") pod \"downloads-76f99584f4-gkl42\" (UID: \"c28672a2-2930-4f95-80b3-0cccb3ce00d9\") " pod="openshift-console/downloads-76f99584f4-gkl42" Sep 16 01:17:18.931043 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.931015 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9s7q\" (UniqueName: \"kubernetes.io/projected/c28672a2-2930-4f95-80b3-0cccb3ce00d9-kube-api-access-n9s7q\") pod \"downloads-76f99584f4-gkl42\" (UID: \"c28672a2-2930-4f95-80b3-0cccb3ce00d9\") " pod="openshift-console/downloads-76f99584f4-gkl42" Sep 16 01:17:18.953099 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:18.953073 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:19.054155 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:19.054123 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-gkl42" Sep 16 01:17:19.222046 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:19.221852 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8"] Sep 16 01:17:19.224322 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:17:19.224299 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9cb20b_07e2_4f91_8693_189a771aaf1c.slice/crio-b8ea36b8cdbfde7eb9da2281886057fa1da7c2f2d73cb75b9f1aa2a539b79247 WatchSource:0}: Error finding container b8ea36b8cdbfde7eb9da2281886057fa1da7c2f2d73cb75b9f1aa2a539b79247: Status 404 returned error can't find the container with id b8ea36b8cdbfde7eb9da2281886057fa1da7c2f2d73cb75b9f1aa2a539b79247 Sep 16 01:17:19.323397 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:19.323231 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-gkl42"] Sep 16 01:17:19.325773 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:17:19.325749 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28672a2_2930_4f95_80b3_0cccb3ce00d9.slice/crio-fdd602b6eb58d8af07b399d2d3c5de45d5aece337f57438e0b0f9468fb083358 WatchSource:0}: Error finding container fdd602b6eb58d8af07b399d2d3c5de45d5aece337f57438e0b0f9468fb083358: Status 404 returned error can't find the container with id fdd602b6eb58d8af07b399d2d3c5de45d5aece337f57438e0b0f9468fb083358 Sep 16 01:17:19.341146 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:19.341114 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-gkl42" event={"ID":"c28672a2-2930-4f95-80b3-0cccb3ce00d9","Type":"ContainerStarted","Data":"fdd602b6eb58d8af07b399d2d3c5de45d5aece337f57438e0b0f9468fb083358"} Sep 16 01:17:19.342191 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:19.342159 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" event={"ID":"2c9cb20b-07e2-4f91-8693-189a771aaf1c","Type":"ContainerStarted","Data":"b8ea36b8cdbfde7eb9da2281886057fa1da7c2f2d73cb75b9f1aa2a539b79247"} Sep 16 01:17:21.743996 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:21.743950 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:17:21.747108 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:21.747073 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ae8823d-2dd3-4b66-a92a-aaca58fad1c6-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-zs9pd\" (UID: \"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:17:21.829031 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:21.829002 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-h8fbk\"" Sep 16 01:17:21.837190 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:21.837058 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" Sep 16 01:17:22.114623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:22.114541 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd"] Sep 16 01:17:22.119543 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:17:22.119507 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae8823d_2dd3_4b66_a92a_aaca58fad1c6.slice/crio-f1e6c7f256f83ca7346145ff4a10568256a6b05ac56cf0cc82a2f18fd21189ae WatchSource:0}: Error finding container f1e6c7f256f83ca7346145ff4a10568256a6b05ac56cf0cc82a2f18fd21189ae: Status 404 returned error can't find the container with id f1e6c7f256f83ca7346145ff4a10568256a6b05ac56cf0cc82a2f18fd21189ae Sep 16 01:17:22.354167 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:22.354124 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" event={"ID":"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6","Type":"ContainerStarted","Data":"f1e6c7f256f83ca7346145ff4a10568256a6b05ac56cf0cc82a2f18fd21189ae"} Sep 16 01:17:23.071201 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.071165 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bdb96c5f5-r5dkg"] Sep 16 01:17:23.074759 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.074735 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.077851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.077829 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 01:17:23.079380 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.078927 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 01:17:23.079380 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.078985 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 01:17:23.079380 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.078932 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 01:17:23.079380 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.079180 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 01:17:23.079380 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.079251 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dwcgh\"" Sep 16 01:17:23.086180 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.086156 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bdb96c5f5-r5dkg"] Sep 16 01:17:23.156643 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.156606 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-oauth-serving-cert\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.156830 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.156679 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-service-ca\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.156830 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.156710 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8n8\" (UniqueName: \"kubernetes.io/projected/d1f58ce5-574e-4879-97f8-578842c82295-kube-api-access-7k8n8\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.156830 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.156803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-console-config\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.156830 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.156830 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-serving-cert\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.157053 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.156848 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-oauth-config\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.257512 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.257472 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-console-config\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.257512 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.257515 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-serving-cert\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.257751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.257534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-oauth-config\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.257751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.257586 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-oauth-serving-cert\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.257751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.257617 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-service-ca\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.257751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.257637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8n8\" (UniqueName: \"kubernetes.io/projected/d1f58ce5-574e-4879-97f8-578842c82295-kube-api-access-7k8n8\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.258388 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.258357 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-console-config\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.258858 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.258824 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-oauth-serving-cert\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.259205 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.259179 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-service-ca\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.261045 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.261005 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-serving-cert\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.261925 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.261889 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-oauth-config\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.265474 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.265450 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8n8\" (UniqueName: \"kubernetes.io/projected/d1f58ce5-574e-4879-97f8-578842c82295-kube-api-access-7k8n8\") pod \"console-7bdb96c5f5-r5dkg\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:23.387646 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:23.387549 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:25.368498 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:25.368420 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" event={"ID":"2c9cb20b-07e2-4f91-8693-189a771aaf1c","Type":"ContainerStarted","Data":"6a2c1052bb3764c5346e3fb8d8504c29d1f0a1d207ea406c075e067dabb27b5a"} Sep 16 01:17:25.368927 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:25.368807 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:25.370459 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:25.370436 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" Sep 16 01:17:25.370644 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:25.370626 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" event={"ID":"9ae8823d-2dd3-4b66-a92a-aaca58fad1c6","Type":"ContainerStarted","Data":"cd3b7a4c472aef0135dd40c5e3368c2c212b164847da567047d7e62825501fca"} Sep 16 01:17:25.379870 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:25.379845 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bdb96c5f5-r5dkg"] Sep 16 01:17:25.384328 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:25.383544 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54f4f89bf7-q9lz8" podStartSLOduration=1.4815530639999999 podStartE2EDuration="7.38352635s" totalImagesPullingTime="5.901973286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:25.382552893 +0000 UTC m=+95.968355105" watchObservedRunningTime="2026-09-16 01:17:25.38352635 +0000 UTC m=+95.969328562" Sep 16 01:17:25.384524 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:17:25.384498 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f58ce5_574e_4879_97f8_578842c82295.slice/crio-b438d8c759175ee2a3ade4ab2935fe646c704022cfecfd6f9beb8ba31819485b WatchSource:0}: Error finding container b438d8c759175ee2a3ade4ab2935fe646c704022cfecfd6f9beb8ba31819485b: Status 404 returned error can't find the container with id b438d8c759175ee2a3ade4ab2935fe646c704022cfecfd6f9beb8ba31819485b Sep 16 01:17:25.401579 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:25.401407 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-zs9pd" podStartSLOduration=33.408681865 podStartE2EDuration="36.401386746s" totalImagesPullingTime="2.992704881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:25.40085502 +0000 UTC m=+95.986657233" watchObservedRunningTime="2026-09-16 01:17:25.401386746 +0000 UTC m=+95.987188956" Sep 16 01:17:26.083782 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:26.083720 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:17:26.086850 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:26.086825 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/799379d2-81b4-458b-9a29-5c43f37c50f1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmhgw\" (UID: \"799379d2-81b4-458b-9a29-5c43f37c50f1\") " pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:17:26.347774 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:26.347678 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kmhgw" Sep 16 01:17:26.381810 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:26.381683 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdb96c5f5-r5dkg" event={"ID":"d1f58ce5-574e-4879-97f8-578842c82295","Type":"ContainerStarted","Data":"b438d8c759175ee2a3ade4ab2935fe646c704022cfecfd6f9beb8ba31819485b"} Sep 16 01:17:26.631245 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:26.631163 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kmhgw"] Sep 16 01:17:26.635909 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:17:26.635874 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799379d2_81b4_458b_9a29_5c43f37c50f1.slice/crio-dfe3895350f5637680e8259d6f225f2d6ecef75ab0e3a3ee3a9d552de2d63bf2 WatchSource:0}: Error finding container dfe3895350f5637680e8259d6f225f2d6ecef75ab0e3a3ee3a9d552de2d63bf2: Status 404 returned error can't find the container with id dfe3895350f5637680e8259d6f225f2d6ecef75ab0e3a3ee3a9d552de2d63bf2 Sep 16 01:17:27.389161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:27.389023 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmhgw" event={"ID":"799379d2-81b4-458b-9a29-5c43f37c50f1","Type":"ContainerStarted","Data":"d670d0d5a6b9b6709e6d72ee22b91b379d363592b435aac8ebffb2d2b5bc95a4"} Sep 16 01:17:27.389161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:27.389091 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmhgw" event={"ID":"799379d2-81b4-458b-9a29-5c43f37c50f1","Type":"ContainerStarted","Data":"dfe3895350f5637680e8259d6f225f2d6ecef75ab0e3a3ee3a9d552de2d63bf2"} Sep 16 01:17:27.698768 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:27.698603 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:17:27.698768 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:27.698676 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:17:27.702543 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:27.702505 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") pod \"image-registry-5d8cdf7994-q99wn\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:17:27.703047 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:27.703022 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f80a113-872c-49ae-b785-75a115bef17a-metrics-tls\") pod \"dns-default-6zdx9\" (UID: \"2f80a113-872c-49ae-b785-75a115bef17a\") " pod="openshift-dns/dns-default-6zdx9" Sep 16 01:17:27.989650 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:27.989614 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-79qw7\"" Sep 16 01:17:27.997699 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:27.997667 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:17:28.002003 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:28.001970 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sq7hb\"" Sep 16 01:17:28.010192 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:28.010121 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6zdx9" Sep 16 01:17:29.366679 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.366656 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6zdx9"] Sep 16 01:17:29.369553 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:17:29.369523 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f80a113_872c_49ae_b785_75a115bef17a.slice/crio-c9810ccd8f6da5f9ac0fea7541b2b48fc7e43d1988c189c9b549e60b10209e4b WatchSource:0}: Error finding container c9810ccd8f6da5f9ac0fea7541b2b48fc7e43d1988c189c9b549e60b10209e4b: Status 404 returned error can't find the container with id c9810ccd8f6da5f9ac0fea7541b2b48fc7e43d1988c189c9b549e60b10209e4b Sep 16 01:17:29.369652 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.369347 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d8cdf7994-q99wn"] Sep 16 01:17:29.371913 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:17:29.371890 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af9fa53_b2cb_4c58_93c3_c790ed0467f6.slice/crio-a6a7467a32b92d25ad3b77b680013f41c56085b74eae9be4b484d9958031cf52 WatchSource:0}: Error finding container a6a7467a32b92d25ad3b77b680013f41c56085b74eae9be4b484d9958031cf52: Status 404 returned error can't find the container with id a6a7467a32b92d25ad3b77b680013f41c56085b74eae9be4b484d9958031cf52 Sep 16 01:17:29.398703 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.398635 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" event={"ID":"3af9fa53-b2cb-4c58-93c3-c790ed0467f6","Type":"ContainerStarted","Data":"a6a7467a32b92d25ad3b77b680013f41c56085b74eae9be4b484d9958031cf52"} Sep 16 01:17:29.401033 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.400996 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmhgw" event={"ID":"799379d2-81b4-458b-9a29-5c43f37c50f1","Type":"ContainerStarted","Data":"7f0282821b86d0fca3a21fbc49819038f03d483f51e8a923c519c99825f14aec"} Sep 16 01:17:29.402473 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.402445 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6zdx9" event={"ID":"2f80a113-872c-49ae-b785-75a115bef17a","Type":"ContainerStarted","Data":"c9810ccd8f6da5f9ac0fea7541b2b48fc7e43d1988c189c9b549e60b10209e4b"} Sep 16 01:17:29.404086 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.404063 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdb96c5f5-r5dkg" event={"ID":"d1f58ce5-574e-4879-97f8-578842c82295","Type":"ContainerStarted","Data":"724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3"} Sep 16 01:17:29.420202 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.420141 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bdb96c5f5-r5dkg" podStartSLOduration=2.706945355 podStartE2EDuration="6.420118864s" totalImagesPullingTime="3.713173509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:29.418849288 +0000 UTC m=+100.004651501" watchObservedRunningTime="2026-09-16 01:17:29.420118864 +0000 UTC m=+100.005921077" Sep 16 01:17:29.691662 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.690620 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl"] Sep 16 01:17:29.709220 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.709162 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl"] Sep 16 01:17:29.709390 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.709319 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.712073 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.712048 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 16 01:17:29.712342 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.712318 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-c87kh\"" Sep 16 01:17:29.712628 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.712610 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 01:17:29.713056 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.713036 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 16 01:17:29.819300 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.819244 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.819456 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.819361 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.819456 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.819401 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgh89\" (UniqueName: \"kubernetes.io/projected/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-kube-api-access-qgh89\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.819456 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.819432 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.920373 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.920333 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.920566 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.920445 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.920566 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.920476 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgh89\" (UniqueName: \"kubernetes.io/projected/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-kube-api-access-qgh89\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.920566 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.920504 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.921199 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.921175 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.923330 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.923301 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.923439 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.923398 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:29.928133 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:29.928113 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgh89\" (UniqueName: \"kubernetes.io/projected/4aab2a54-aac1-4a3a-970c-2fd77e17bdeb-kube-api-access-qgh89\") pod \"prometheus-operator-68bcd7f987-pjvgl\" (UID: \"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:30.021661 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:30.021618 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" Sep 16 01:17:30.409152 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:30.409055 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" event={"ID":"3af9fa53-b2cb-4c58-93c3-c790ed0467f6","Type":"ContainerStarted","Data":"4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da"} Sep 16 01:17:30.409584 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:30.409258 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:17:30.429123 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:30.429072 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" podStartSLOduration=100.429057008 podStartE2EDuration="1m40.429057008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:30.427429617 +0000 UTC m=+101.013231840" watchObservedRunningTime="2026-09-16 01:17:30.429057008 +0000 UTC m=+101.014859217" Sep 16 01:17:31.284989 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:31.284950 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xgkwn" Sep 16 01:17:33.388563 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:33.388525 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:33.389113 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:33.388613 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:17:33.390260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:33.390230 2567 patch_prober.go:28] interesting pod/console-7bdb96c5f5-r5dkg container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.15:8443/health\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body= Sep 16 01:17:33.390402 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:33.390296 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7bdb96c5f5-r5dkg" podUID="d1f58ce5-574e-4879-97f8-578842c82295" containerName="console" probeResult="failure" output="Get \"https://10.132.0.15:8443/health\": dial tcp 10.132.0.15:8443: connect: connection refused" Sep 16 01:17:36.815693 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:36.815617 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl"] Sep 16 01:17:37.443193 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:37.443104 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-gkl42" event={"ID":"c28672a2-2930-4f95-80b3-0cccb3ce00d9","Type":"ContainerStarted","Data":"1fa92d738c400c7c54c128a05547999407d5c826628796dcf8189f443e16a5c3"} Sep 16 01:17:37.443677 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:37.443628 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-gkl42" Sep 16 01:17:37.447510 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:37.447472 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" event={"ID":"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb","Type":"ContainerStarted","Data":"e8c618c8cde7ff590857ed03c03890b5138b6e9e67dbabba60aed49b55ff5e43"} Sep 16 01:17:37.456945 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:37.456869 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-gkl42" Sep 16 01:17:37.461934 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:37.460499 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-gkl42" podStartSLOduration=2.159105421 podStartE2EDuration="19.460483701s" totalImagesPullingTime="17.30137828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:37.459098924 +0000 UTC m=+108.044901138" watchObservedRunningTime="2026-09-16 01:17:37.460483701 +0000 UTC m=+108.046285942" Sep 16 01:17:39.460979 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:39.460949 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmhgw" event={"ID":"799379d2-81b4-458b-9a29-5c43f37c50f1","Type":"ContainerStarted","Data":"7e2f3bad3eeb7d8e5dfe6bdd408214661d198360d389a9918e94b2a2cfde5946"} Sep 16 01:17:39.463191 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:39.463158 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6zdx9" event={"ID":"2f80a113-872c-49ae-b785-75a115bef17a","Type":"ContainerStarted","Data":"366507f02bd62b0389a6d1bc711b6e3f986768dff14e26a1980fdf004ba003ad"} Sep 16 01:17:39.477325 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:39.477243 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kmhgw" podStartSLOduration=33.044320575 podStartE2EDuration="45.47722604s" totalImagesPullingTime="12.432905465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:16:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:39.476844252 +0000 UTC m=+110.062646466" watchObservedRunningTime="2026-09-16 01:17:39.47722604 +0000 UTC m=+110.063028266" Sep 16 01:17:40.468826 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:40.468781 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" event={"ID":"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb","Type":"ContainerStarted","Data":"f93fa3d31728d63cd0a320011ca3ec81180815f8d75a309d41e9a0a48c358b52"} Sep 16 01:17:40.469418 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:40.468845 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" event={"ID":"4aab2a54-aac1-4a3a-970c-2fd77e17bdeb","Type":"ContainerStarted","Data":"41ca75068673b84d8c7405b40f0ada90c7865ff75b796b99ecffae6d1255079c"} Sep 16 01:17:40.471031 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:40.471001 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6zdx9" event={"ID":"2f80a113-872c-49ae-b785-75a115bef17a","Type":"ContainerStarted","Data":"bee7cd4f8534fabbb5dbcc0a051c3040fcd0d9b1ea744fbd780185db066e6efa"} Sep 16 01:17:40.471141 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:40.471098 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6zdx9" Sep 16 01:17:40.485635 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:40.485589 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pjvgl" podStartSLOduration=8.723286854 podStartE2EDuration="11.485575402s" totalImagesPullingTime="2.762288548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:40.483373327 +0000 UTC m=+111.069175542" watchObservedRunningTime="2026-09-16 01:17:40.485575402 +0000 UTC m=+111.071377613" Sep 16 01:17:40.499022 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:40.498984 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6zdx9" podStartSLOduration=67.558508121 podStartE2EDuration="1m17.498969913s" totalImagesPullingTime="9.940461792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:40.497858716 +0000 UTC m=+111.083660926" watchObservedRunningTime="2026-09-16 01:17:40.498969913 +0000 UTC m=+111.084772143" Sep 16 01:17:41.838970 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:41.838936 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bdb96c5f5-r5dkg"] Sep 16 01:17:42.034665 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.034621 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk"] Sep 16 01:17:42.054601 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.054557 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk"] Sep 16 01:17:42.055869 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.054901 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.079889 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.079854 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r2pw7"] Sep 16 01:17:42.081015 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.080993 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 16 01:17:42.083033 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.081196 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 16 01:17:42.083033 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.081442 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vfbrr\"" Sep 16 01:17:42.099081 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.098995 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.103183 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.103154 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 01:17:42.103318 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.103209 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 01:17:42.103482 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.103459 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bnfrk\"" Sep 16 01:17:42.103592 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.103468 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 01:17:42.240672 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.240628 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-textfile\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.240866 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.240685 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-wtmp\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.240866 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.240718 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.240866 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.240805 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84bc2eeb-73e5-4859-ae77-6a8543135f2c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.241045 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.240879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccf8013-4766-4267-87e7-735069220a4e-metrics-client-ca\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.241045 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.240923 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.241045 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.240959 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.241045 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.241010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-tls\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.241045 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.241035 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-accelerators-collector-config\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.241272 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.241058 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxpvx\" (UniqueName: \"kubernetes.io/projected/cccf8013-4766-4267-87e7-735069220a4e-kube-api-access-dxpvx\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.241272 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.241088 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5clpt\" (UniqueName: \"kubernetes.io/projected/84bc2eeb-73e5-4859-ae77-6a8543135f2c-kube-api-access-5clpt\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.241272 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.241110 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cccf8013-4766-4267-87e7-735069220a4e-sys\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.241272 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.241135 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cccf8013-4766-4267-87e7-735069220a4e-root\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.342296 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342055 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-textfile\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.342296 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342101 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-wtmp\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.342296 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342134 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.342296 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342168 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84bc2eeb-73e5-4859-ae77-6a8543135f2c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.342296 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342198 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccf8013-4766-4267-87e7-735069220a4e-metrics-client-ca\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.342296 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342225 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.342296 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342263 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.342753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-tls\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.342753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342371 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-accelerators-collector-config\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.342753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxpvx\" (UniqueName: \"kubernetes.io/projected/cccf8013-4766-4267-87e7-735069220a4e-kube-api-access-dxpvx\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.342753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342421 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5clpt\" (UniqueName: \"kubernetes.io/projected/84bc2eeb-73e5-4859-ae77-6a8543135f2c-kube-api-access-5clpt\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.342753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342439 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cccf8013-4766-4267-87e7-735069220a4e-sys\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.342753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342462 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cccf8013-4766-4267-87e7-735069220a4e-root\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.342753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342552 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cccf8013-4766-4267-87e7-735069220a4e-root\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.343093 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.342920 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-textfile\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.346290 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.343705 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-wtmp\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.346290 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:42.343793 2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 16 01:17:42.346290 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:42.343851 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-tls podName:84bc2eeb-73e5-4859-ae77-6a8543135f2c nodeName:}" failed. No retries permitted until 2026-09-16 01:17:42.843833323 +0000 UTC m=+113.429635514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-zk5tk" (UID: "84bc2eeb-73e5-4859-ae77-6a8543135f2c") : secret "openshift-state-metrics-tls" not found Sep 16 01:17:42.346290 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.344147 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccf8013-4766-4267-87e7-735069220a4e-metrics-client-ca\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.346290 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.344685 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84bc2eeb-73e5-4859-ae77-6a8543135f2c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.346290 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.344933 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cccf8013-4766-4267-87e7-735069220a4e-sys\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.346290 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.343705 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-accelerators-collector-config\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.346290 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.345733 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.349322 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.347537 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-tls\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.359293 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.356270 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5clpt\" (UniqueName: \"kubernetes.io/projected/84bc2eeb-73e5-4859-ae77-6a8543135f2c-kube-api-access-5clpt\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.359293 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.357041 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cccf8013-4766-4267-87e7-735069220a4e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.365102 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.365073 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxpvx\" (UniqueName: \"kubernetes.io/projected/cccf8013-4766-4267-87e7-735069220a4e-kube-api-access-dxpvx\") pod \"node-exporter-r2pw7\" (UID: \"cccf8013-4766-4267-87e7-735069220a4e\") " pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.414221 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.414174 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r2pw7" Sep 16 01:17:42.491657 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.491008 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2pw7" event={"ID":"cccf8013-4766-4267-87e7-735069220a4e","Type":"ContainerStarted","Data":"14ee04f9546398219b88912574e9be178fd02c75e634345e5310eb645ac5c715"} Sep 16 01:17:42.847911 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:42.847811 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:42.848415 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:42.847921 2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 16 01:17:42.848415 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:42.847996 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-tls podName:84bc2eeb-73e5-4859-ae77-6a8543135f2c nodeName:}" failed. No retries permitted until 2026-09-16 01:17:43.847976358 +0000 UTC m=+114.433778549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-zk5tk" (UID: "84bc2eeb-73e5-4859-ae77-6a8543135f2c") : secret "openshift-state-metrics-tls" not found Sep 16 01:17:43.122498 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.120474 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 01:17:43.160443 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.160353 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 01:17:43.160624 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.160523 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.165515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.163262 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 16 01:17:43.165515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.163597 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 16 01:17:43.165515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.163647 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 16 01:17:43.165515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.164169 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 16 01:17:43.165515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.164391 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 16 01:17:43.165515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.164580 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 16 01:17:43.165515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.164763 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 16 01:17:43.165515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.164946 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 16 01:17:43.165515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.165145 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 16 01:17:43.165515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.165364 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tkbtl\"" Sep 16 01:17:43.352895 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.352857 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.352895 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.352909 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c509b351-15b1-4336-866c-12094e7ef184-config-out\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.353153 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.352943 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.353153 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.352994 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c509b351-15b1-4336-866c-12094e7ef184-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.353153 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.353055 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c509b351-15b1-4336-866c-12094e7ef184-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.353153 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.353087 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.353377 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.353170 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.353377 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.353206 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c509b351-15b1-4336-866c-12094e7ef184-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.353377 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.353234 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-web-config\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.353377 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.353297 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-config-volume\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.353377 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.353357 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c509b351-15b1-4336-866c-12094e7ef184-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.353613 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.353380 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctlh2\" (UniqueName: \"kubernetes.io/projected/c509b351-15b1-4336-866c-12094e7ef184-kube-api-access-ctlh2\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.353613 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.353425 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454289 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c509b351-15b1-4336-866c-12094e7ef184-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454343 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c509b351-15b1-4336-866c-12094e7ef184-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454454 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-web-config\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454493 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-config-volume\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454543 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c509b351-15b1-4336-866c-12094e7ef184-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454569 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctlh2\" (UniqueName: \"kubernetes.io/projected/c509b351-15b1-4336-866c-12094e7ef184-kube-api-access-ctlh2\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454608 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454648 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454674 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c509b351-15b1-4336-866c-12094e7ef184-config-out\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454698 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.455175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.454723 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c509b351-15b1-4336-866c-12094e7ef184-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.460191 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.457466 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c509b351-15b1-4336-866c-12094e7ef184-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.460191 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.459864 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c509b351-15b1-4336-866c-12094e7ef184-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.460418 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.460333 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c509b351-15b1-4336-866c-12094e7ef184-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.465387 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.465325 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-web-config\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.466298 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.465853 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.466298 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.465928 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c509b351-15b1-4336-866c-12094e7ef184-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.470505 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.470458 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.471140 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.471042 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c509b351-15b1-4336-866c-12094e7ef184-config-out\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.472527 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.472470 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.477563 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.477493 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-config-volume\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.478067 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.478034 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.478456 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.478427 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c509b351-15b1-4336-866c-12094e7ef184-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.484108 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.484086 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctlh2\" (UniqueName: \"kubernetes.io/projected/c509b351-15b1-4336-866c-12094e7ef184-kube-api-access-ctlh2\") pod \"alertmanager-main-0\" (UID: \"c509b351-15b1-4336-866c-12094e7ef184\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.776791 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.776748 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 01:17:43.861334 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.860416 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:43.864533 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.864476 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84bc2eeb-73e5-4859-ae77-6a8543135f2c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-zk5tk\" (UID: \"84bc2eeb-73e5-4859-ae77-6a8543135f2c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:43.880136 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:43.878444 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" Sep 16 01:17:44.097907 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:44.097878 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 01:17:44.159545 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:17:44.159510 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc509b351_15b1_4336_866c_12094e7ef184.slice/crio-f4979b290a8d8709bce303ecd0df1628dc9d3632ea43ca2f6ea6165aa3248e23 WatchSource:0}: Error finding container f4979b290a8d8709bce303ecd0df1628dc9d3632ea43ca2f6ea6165aa3248e23: Status 404 returned error can't find the container with id f4979b290a8d8709bce303ecd0df1628dc9d3632ea43ca2f6ea6165aa3248e23 Sep 16 01:17:44.179384 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:44.179348 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk"] Sep 16 01:17:44.183887 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:17:44.183854 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bc2eeb_73e5_4859_ae77_6a8543135f2c.slice/crio-ed69a61a011486f1e7fca382bbdf4e4bc49704f6c2fe321b746ceddee5deb882 WatchSource:0}: Error finding container ed69a61a011486f1e7fca382bbdf4e4bc49704f6c2fe321b746ceddee5deb882: Status 404 returned error can't find the container with id ed69a61a011486f1e7fca382bbdf4e4bc49704f6c2fe321b746ceddee5deb882 Sep 16 01:17:44.501657 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:44.501621 2567 generic.go:356] "Generic (PLEG): container finished" podID="cccf8013-4766-4267-87e7-735069220a4e" containerID="851b7d5658a86087127fc3838ba871cf0c34a102e336e9a9fe0e214177c5d0d3" exitCode=0 Sep 16 01:17:44.501816 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:44.501712 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2pw7" event={"ID":"cccf8013-4766-4267-87e7-735069220a4e","Type":"ContainerDied","Data":"851b7d5658a86087127fc3838ba871cf0c34a102e336e9a9fe0e214177c5d0d3"} Sep 16 01:17:44.503316 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:44.503270 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c509b351-15b1-4336-866c-12094e7ef184","Type":"ContainerStarted","Data":"f4979b290a8d8709bce303ecd0df1628dc9d3632ea43ca2f6ea6165aa3248e23"} Sep 16 01:17:44.505625 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:44.505600 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" event={"ID":"84bc2eeb-73e5-4859-ae77-6a8543135f2c","Type":"ContainerStarted","Data":"d2cbbbfdf02087f62116e15bf5a36f0e6bd330eaef6743f245ab311591cb7396"} Sep 16 01:17:44.505724 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:44.505653 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" event={"ID":"84bc2eeb-73e5-4859-ae77-6a8543135f2c","Type":"ContainerStarted","Data":"74d1423317a24158e8f0bc8a7dc5ba8445e84215fdfbcc20a0f9a094145d6474"} Sep 16 01:17:44.505724 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:44.505682 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" event={"ID":"84bc2eeb-73e5-4859-ae77-6a8543135f2c","Type":"ContainerStarted","Data":"ed69a61a011486f1e7fca382bbdf4e4bc49704f6c2fe321b746ceddee5deb882"} Sep 16 01:17:44.761132 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:44.761092 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d8cdf7994-q99wn"] Sep 16 01:17:44.769102 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:44.769034 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:17:45.512761 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:45.512720 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2pw7" event={"ID":"cccf8013-4766-4267-87e7-735069220a4e","Type":"ContainerStarted","Data":"b8468fae6716cbcc684d87f8ca2c79f57fbb3cd44be02c91fc78e30a68467f15"} Sep 16 01:17:45.513153 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:45.512832 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r2pw7" event={"ID":"cccf8013-4766-4267-87e7-735069220a4e","Type":"ContainerStarted","Data":"00f6f3ef0880f38487884d09ba87aaf92a21de5a450f63ab6699daf88da12b1f"} Sep 16 01:17:45.532259 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:45.532212 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r2pw7" podStartSLOduration=1.714295608 podStartE2EDuration="3.532196582s" totalImagesPullingTime="1.719817694s" totalInitContainerRuntime="98.08328ms" isStatefulPod=true podCreationTimestamp="2026-09-16 01:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:45.530019466 +0000 UTC m=+116.115821677" watchObservedRunningTime="2026-09-16 01:17:45.532196582 +0000 UTC m=+116.117998787" Sep 16 01:17:46.518741 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:46.518705 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" event={"ID":"84bc2eeb-73e5-4859-ae77-6a8543135f2c","Type":"ContainerStarted","Data":"33f61cc91a61b303f0a371496422c4d7bfaa32877ffae8e36a9595163d5d923b"} Sep 16 01:17:46.520265 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:46.520241 2567 generic.go:356] "Generic (PLEG): container finished" podID="c509b351-15b1-4336-866c-12094e7ef184" containerID="b44d0bcef08c0750a2de5625b54280a715a8f0e0753f35e5b3805a51d68639e1" exitCode=0 Sep 16 01:17:46.520423 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:46.520307 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c509b351-15b1-4336-866c-12094e7ef184","Type":"ContainerDied","Data":"b44d0bcef08c0750a2de5625b54280a715a8f0e0753f35e5b3805a51d68639e1"} Sep 16 01:17:46.536170 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:46.536125 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-zk5tk" podStartSLOduration=2.7756049640000002 podStartE2EDuration="4.536111225s" totalImagesPullingTime="1.760506261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:46.534873645 +0000 UTC m=+117.120675850" watchObservedRunningTime="2026-09-16 01:17:46.536111225 +0000 UTC m=+117.121913435" Sep 16 01:17:48.259929 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.259894 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 01:17:48.279263 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.279227 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 01:17:48.279463 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.279381 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.282040 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.282012 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9slz7\"" Sep 16 01:17:48.282190 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.282073 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 16 01:17:48.282371 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.282353 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 16 01:17:48.282510 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.282493 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 16 01:17:48.283442 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.283420 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 16 01:17:48.284914 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.283631 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 16 01:17:48.284914 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.283468 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 16 01:17:48.284914 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.284504 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 16 01:17:48.285378 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.285126 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 16 01:17:48.286023 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.285510 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 16 01:17:48.286023 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.285771 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 16 01:17:48.290663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.286768 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 16 01:17:48.290663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.287008 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 01:17:48.290663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.288097 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 16 01:17:48.290663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.288203 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 16 01:17:48.290663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.289852 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5vs90cufam1p4\"" Sep 16 01:17:48.292960 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.292904 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 16 01:17:48.300962 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.300942 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301055 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.300972 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-config\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301055 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.300989 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301055 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301055 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301037 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301209 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301067 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301209 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301107 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301209 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301131 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7wtw\" (UniqueName: \"kubernetes.io/projected/950bc2d7-ca5b-476a-875f-f055436c65fd-kube-api-access-k7wtw\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301209 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301150 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301209 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301167 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301403 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301226 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301403 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301308 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301403 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301335 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301403 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301372 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/950bc2d7-ca5b-476a-875f-f055436c65fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301533 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301440 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301533 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301464 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301533 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301506 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/950bc2d7-ca5b-476a-875f-f055436c65fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301621 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301550 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301621 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301579 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.301621 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.301613 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.402886 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.402843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.402886 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.402889 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.403134 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.402910 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.403134 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.402937 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.403134 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.402955 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.403134 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.402974 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/950bc2d7-ca5b-476a-875f-f055436c65fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.403134 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.403006 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.403134 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.403030 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.403134 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.403080 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/950bc2d7-ca5b-476a-875f-f055436c65fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.403134 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.403106 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.403743 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.403805 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.403845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.403909 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.403942 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-config\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.403966 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.404010 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.404035 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.404057 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.404088 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.404234 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7wtw\" (UniqueName: \"kubernetes.io/projected/950bc2d7-ca5b-476a-875f-f055436c65fd-kube-api-access-k7wtw\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.404479 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:17:48.404479 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-trusted-ca-bundle podName:950bc2d7-ca5b-476a-875f-f055436c65fd nodeName:}" failed. No retries permitted until 2026-09-16 01:17:48.904458187 +0000 UTC m=+119.490260379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-trusted-ca-bundle") pod "prometheus-k8s-0" (UID: "950bc2d7-ca5b-476a-875f-f055436c65fd") : configmap references non-existent config key: ca-bundle.crt Sep 16 01:17:48.405165 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.404750 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.405334 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.405307 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.405769 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.405745 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.406704 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.406678 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/950bc2d7-ca5b-476a-875f-f055436c65fd-config-out\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.406801 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.406687 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.408254 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.407470 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/950bc2d7-ca5b-476a-875f-f055436c65fd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.408254 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.407704 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.408965 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.408918 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.410739 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.410719 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.410950 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.410910 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.411068 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.411021 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.411190 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.411170 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.411465 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.411444 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.411676 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.411655 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.413495 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.413456 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-web-config\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.413711 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.413670 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7wtw\" (UniqueName: \"kubernetes.io/projected/950bc2d7-ca5b-476a-875f-f055436c65fd-kube-api-access-k7wtw\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.414623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.414587 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-config\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.414831 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.414807 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/950bc2d7-ca5b-476a-875f-f055436c65fd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.908751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.908721 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:48.909536 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:48.909511 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/950bc2d7-ca5b-476a-875f-f055436c65fd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"950bc2d7-ca5b-476a-875f-f055436c65fd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:49.199491 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:49.199451 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:49.494443 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:49.494414 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 01:17:49.498415 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:17:49.498386 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950bc2d7_ca5b_476a_875f_f055436c65fd.slice/crio-fd05f729024758f8469748f5e9fba16571e2c8c9c7645f8301849f76f5b264f1 WatchSource:0}: Error finding container fd05f729024758f8469748f5e9fba16571e2c8c9c7645f8301849f76f5b264f1: Status 404 returned error can't find the container with id fd05f729024758f8469748f5e9fba16571e2c8c9c7645f8301849f76f5b264f1 Sep 16 01:17:49.533400 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:49.533364 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"950bc2d7-ca5b-476a-875f-f055436c65fd","Type":"ContainerStarted","Data":"fd05f729024758f8469748f5e9fba16571e2c8c9c7645f8301849f76f5b264f1"} Sep 16 01:17:49.536624 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:49.536602 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c509b351-15b1-4336-866c-12094e7ef184","Type":"ContainerStarted","Data":"3e228220ed0401b145d23689994a2aab369b935e947098ae9b6666ea1783a2ec"} Sep 16 01:17:49.536694 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:49.536664 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c509b351-15b1-4336-866c-12094e7ef184","Type":"ContainerStarted","Data":"eb23b05fa46468f2e782ffb699249e064914ed94c841d0019836c52cdaa94809"} Sep 16 01:17:49.536694 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:49.536692 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c509b351-15b1-4336-866c-12094e7ef184","Type":"ContainerStarted","Data":"b6f3103465072908c5df9a8bd59b1884f41c8153f2bc24adf875019be1c7cfd7"} Sep 16 01:17:49.536771 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:49.536716 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c509b351-15b1-4336-866c-12094e7ef184","Type":"ContainerStarted","Data":"aa24a8da1ffc2c9c895c70b07cbfbe01405fb4f025d4dfd84c203c6fb472cdf3"} Sep 16 01:17:49.536771 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:49.536739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c509b351-15b1-4336-866c-12094e7ef184","Type":"ContainerStarted","Data":"8cba1a11faa7c3b66574876f31cf9e04461de94ee90a21fe1e96fe6a29aea511"} Sep 16 01:17:50.477339 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:50.477304 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6zdx9" Sep 16 01:17:50.542344 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:50.542315 2567 generic.go:356] "Generic (PLEG): container finished" podID="950bc2d7-ca5b-476a-875f-f055436c65fd" containerID="750a87c2016db228907d0c165901758e2952b5c7a6f0bdef6693a84faddc73cb" exitCode=0 Sep 16 01:17:50.542753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:50.542397 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"950bc2d7-ca5b-476a-875f-f055436c65fd","Type":"ContainerDied","Data":"750a87c2016db228907d0c165901758e2952b5c7a6f0bdef6693a84faddc73cb"} Sep 16 01:17:51.553782 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:51.553741 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c509b351-15b1-4336-866c-12094e7ef184","Type":"ContainerStarted","Data":"5a1a68841a05242554c7ae924bf7d0611927f34a62fdac52ac852eeeaf4bedf5"} Sep 16 01:17:51.580481 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:51.580428 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.053720191 podStartE2EDuration="8.580399461s" totalImagesPullingTime="5.504642501s" totalInitContainerRuntime="22.036769ms" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:51.578407029 +0000 UTC m=+122.164209240" watchObservedRunningTime="2026-09-16 01:17:51.580399461 +0000 UTC m=+122.166201673" Sep 16 01:17:55.573645 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:55.573607 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"950bc2d7-ca5b-476a-875f-f055436c65fd","Type":"ContainerStarted","Data":"b37fcb7096f0cec4020df2e7862d70c812c8b2916e1ed068a22c7231ff75b13e"} Sep 16 01:17:55.574131 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:55.573719 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"950bc2d7-ca5b-476a-875f-f055436c65fd","Type":"ContainerStarted","Data":"5f23796ffc28051dbc892a8d6aefc462bbbd62044fd30796e5b89dfcfb546314"} Sep 16 01:17:57.586048 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:57.586018 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"950bc2d7-ca5b-476a-875f-f055436c65fd","Type":"ContainerStarted","Data":"c0561e798d1a0b3a015b1dddf4d4d0d37e4e169fd8ee6dda59257ad2341cb7b5"} Sep 16 01:17:57.586408 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:57.586105 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"950bc2d7-ca5b-476a-875f-f055436c65fd","Type":"ContainerStarted","Data":"eb711f29ba9af0fd1d1f2dbb7e2c46f1ca1f2b6d2905919fa7ff6eb051f93ebe"} Sep 16 01:17:58.592945 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:58.592909 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"950bc2d7-ca5b-476a-875f-f055436c65fd","Type":"ContainerStarted","Data":"a4c34a5e984e272218ee1a8b956801b1c03335136f6c2f3883ac214464513857"} Sep 16 01:17:58.593372 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:58.592970 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"950bc2d7-ca5b-476a-875f-f055436c65fd","Type":"ContainerStarted","Data":"db67e584f860460cb5259db17079b1c701685feaebef0065f3c730e7ebbfc9ca"} Sep 16 01:17:58.622798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:58.622742 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.127538777 podStartE2EDuration="10.622724625s" totalImagesPullingTime="6.476728172s" totalInitContainerRuntime="18.457676ms" isStatefulPod=false podCreationTimestamp="2026-09-16 01:17:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:17:58.620514402 +0000 UTC m=+129.206316612" watchObservedRunningTime="2026-09-16 01:17:58.622724625 +0000 UTC m=+129.208526837" Sep 16 01:17:58.805211 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:58.805175 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:17:58.807691 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:58.807654 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00728728-c5dc-4f6c-b96b-8356103fafa5-metrics-certs\") pod \"network-metrics-daemon-snt2b\" (UID: \"00728728-c5dc-4f6c-b96b-8356103fafa5\") " pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:17:58.847102 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:58.847023 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w6h4t\"" Sep 16 01:17:58.855137 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:58.855108 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-snt2b" Sep 16 01:17:59.144770 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:59.144743 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-snt2b"] Sep 16 01:17:59.146999 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:17:59.146971 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00728728_c5dc_4f6c_b96b_8356103fafa5.slice/crio-9761e9530101b8c9e95b7b111cddfc6c3002d749fcfe81ba85ff99c7bed09782 WatchSource:0}: Error finding container 9761e9530101b8c9e95b7b111cddfc6c3002d749fcfe81ba85ff99c7bed09782: Status 404 returned error can't find the container with id 9761e9530101b8c9e95b7b111cddfc6c3002d749fcfe81ba85ff99c7bed09782 Sep 16 01:17:59.199975 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:59.199948 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:17:59.597517 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:17:59.597476 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-snt2b" event={"ID":"00728728-c5dc-4f6c-b96b-8356103fafa5","Type":"ContainerStarted","Data":"9761e9530101b8c9e95b7b111cddfc6c3002d749fcfe81ba85ff99c7bed09782"} Sep 16 01:18:01.613429 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:01.613382 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-snt2b" event={"ID":"00728728-c5dc-4f6c-b96b-8356103fafa5","Type":"ContainerStarted","Data":"229f779343db2882d12f7a75c73476e4d4353bc5148e5348825697fd787558b9"} Sep 16 01:18:01.613822 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:01.613482 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-snt2b" event={"ID":"00728728-c5dc-4f6c-b96b-8356103fafa5","Type":"ContainerStarted","Data":"0bf9df6f284914690c06fafe02789b5b222a8bcc901daccae1b9af15e756273f"} Sep 16 01:18:01.629485 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:01.629433 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-snt2b" podStartSLOduration=130.207138161 podStartE2EDuration="2m11.629417552s" totalImagesPullingTime="1.422279391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:18:01.627683756 +0000 UTC m=+132.213485967" watchObservedRunningTime="2026-09-16 01:18:01.629417552 +0000 UTC m=+132.215219763" Sep 16 01:18:06.863161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:06.863113 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bdb96c5f5-r5dkg" podUID="d1f58ce5-574e-4879-97f8-578842c82295" containerName="console" containerID="cri-o://724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3" gracePeriod=15 Sep 16 01:18:07.099993 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.099969 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:18:07.179068 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.178984 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-serving-cert\") pod \"d1f58ce5-574e-4879-97f8-578842c82295\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " Sep 16 01:18:07.179068 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.179022 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-oauth-serving-cert\") pod \"d1f58ce5-574e-4879-97f8-578842c82295\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " Sep 16 01:18:07.179068 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.179041 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k8n8\" (UniqueName: \"kubernetes.io/projected/d1f58ce5-574e-4879-97f8-578842c82295-kube-api-access-7k8n8\") pod \"d1f58ce5-574e-4879-97f8-578842c82295\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " Sep 16 01:18:07.179314 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.179075 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-service-ca\") pod \"d1f58ce5-574e-4879-97f8-578842c82295\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " Sep 16 01:18:07.179314 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.179122 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-console-config\") pod \"d1f58ce5-574e-4879-97f8-578842c82295\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " Sep 16 01:18:07.179314 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.179147 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-oauth-config\") pod \"d1f58ce5-574e-4879-97f8-578842c82295\" (UID: \"d1f58ce5-574e-4879-97f8-578842c82295\") " Sep 16 01:18:07.179439 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.179401 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d1f58ce5-574e-4879-97f8-578842c82295" (UID: "d1f58ce5-574e-4879-97f8-578842c82295"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:18:07.179564 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.179536 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-console-config" (OuterVolumeSpecName: "console-config") pod "d1f58ce5-574e-4879-97f8-578842c82295" (UID: "d1f58ce5-574e-4879-97f8-578842c82295"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:18:07.179564 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.179547 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-service-ca" (OuterVolumeSpecName: "service-ca") pod "d1f58ce5-574e-4879-97f8-578842c82295" (UID: "d1f58ce5-574e-4879-97f8-578842c82295"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:18:07.181369 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.181345 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d1f58ce5-574e-4879-97f8-578842c82295" (UID: "d1f58ce5-574e-4879-97f8-578842c82295"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:18:07.181464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.181366 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d1f58ce5-574e-4879-97f8-578842c82295" (UID: "d1f58ce5-574e-4879-97f8-578842c82295"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:18:07.181565 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.181544 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f58ce5-574e-4879-97f8-578842c82295-kube-api-access-7k8n8" (OuterVolumeSpecName: "kube-api-access-7k8n8") pod "d1f58ce5-574e-4879-97f8-578842c82295" (UID: "d1f58ce5-574e-4879-97f8-578842c82295"). InnerVolumeSpecName "kube-api-access-7k8n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:18:07.280750 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.280717 2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-serving-cert\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:07.280750 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.280744 2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-oauth-serving-cert\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:07.280750 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.280753 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k8n8\" (UniqueName: \"kubernetes.io/projected/d1f58ce5-574e-4879-97f8-578842c82295-kube-api-access-7k8n8\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:07.280979 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.280763 2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:07.280979 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.280772 2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d1f58ce5-574e-4879-97f8-578842c82295-console-config\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:07.280979 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.280780 2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d1f58ce5-574e-4879-97f8-578842c82295-console-oauth-config\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:07.636408 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.636378 2567 generic.go:356] "Generic (PLEG): container finished" podID="d1f58ce5-574e-4879-97f8-578842c82295" containerID="724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3" exitCode=0 Sep 16 01:18:07.636569 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.636424 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bdb96c5f5-r5dkg" Sep 16 01:18:07.636569 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.636467 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdb96c5f5-r5dkg" event={"ID":"d1f58ce5-574e-4879-97f8-578842c82295","Type":"ContainerDied","Data":"724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3"} Sep 16 01:18:07.636569 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.636536 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bdb96c5f5-r5dkg" event={"ID":"d1f58ce5-574e-4879-97f8-578842c82295","Type":"ContainerDied","Data":"b438d8c759175ee2a3ade4ab2935fe646c704022cfecfd6f9beb8ba31819485b"} Sep 16 01:18:07.636569 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.636562 2567 scope.go:117] "RemoveContainer" containerID="724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3" Sep 16 01:18:07.649271 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.649251 2567 scope.go:117] "RemoveContainer" containerID="724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3" Sep 16 01:18:07.649622 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:18:07.649601 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3\": container with ID starting with 724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3 not found: ID does not exist" containerID="724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3" Sep 16 01:18:07.649675 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.649630 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3"} err="failed to get container status \"724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3\": rpc error: code = NotFound desc = could not find container \"724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3\": container with ID starting with 724aa1fb46461a504ba526ab319c29d4ae7d02b632e723fba045e59f0faf56c3 not found: ID does not exist" Sep 16 01:18:07.659819 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.659795 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bdb96c5f5-r5dkg"] Sep 16 01:18:07.662141 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.662117 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bdb96c5f5-r5dkg"] Sep 16 01:18:07.938585 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:07.938509 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1f58ce5-574e-4879-97f8-578842c82295" path="/var/lib/kubelet/pods/d1f58ce5-574e-4879-97f8-578842c82295/volumes" Sep 16 01:18:09.793013 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:09.792943 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" podUID="3af9fa53-b2cb-4c58-93c3-c790ed0467f6" containerName="registry" containerID="cri-o://4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da" gracePeriod=30 Sep 16 01:18:10.035404 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.035380 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:18:10.108042 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.107913 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-trusted-ca\") pod \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " Sep 16 01:18:10.108042 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.107948 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-ca-trust-extracted\") pod \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " Sep 16 01:18:10.108042 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.107982 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdqvf\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-kube-api-access-vdqvf\") pod \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " Sep 16 01:18:10.108042 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.108004 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-installation-pull-secrets\") pod \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " Sep 16 01:18:10.108042 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.108045 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-certificates\") pod \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " Sep 16 01:18:10.108445 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.108085 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") pod \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " Sep 16 01:18:10.108445 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.108125 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-bound-sa-token\") pod \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " Sep 16 01:18:10.108445 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.108160 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-image-registry-private-configuration\") pod \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\" (UID: \"3af9fa53-b2cb-4c58-93c3-c790ed0467f6\") " Sep 16 01:18:10.108595 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.108566 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3af9fa53-b2cb-4c58-93c3-c790ed0467f6" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:18:10.108994 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.108885 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3af9fa53-b2cb-4c58-93c3-c790ed0467f6" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:18:10.110819 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.110791 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3af9fa53-b2cb-4c58-93c3-c790ed0467f6" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:18:10.110928 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.110828 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3af9fa53-b2cb-4c58-93c3-c790ed0467f6" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:18:10.110928 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.110867 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3af9fa53-b2cb-4c58-93c3-c790ed0467f6" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:18:10.111019 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.110926 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-kube-api-access-vdqvf" (OuterVolumeSpecName: "kube-api-access-vdqvf") pod "3af9fa53-b2cb-4c58-93c3-c790ed0467f6" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6"). InnerVolumeSpecName "kube-api-access-vdqvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:18:10.111086 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.111066 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3af9fa53-b2cb-4c58-93c3-c790ed0467f6" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:18:10.116484 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.116460 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3af9fa53-b2cb-4c58-93c3-c790ed0467f6" (UID: "3af9fa53-b2cb-4c58-93c3-c790ed0467f6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:18:10.209769 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.209731 2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-installation-pull-secrets\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:10.209769 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.209764 2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-certificates\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:10.209961 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.209777 2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-registry-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:10.209961 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.209789 2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-bound-sa-token\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:10.209961 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.209801 2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-image-registry-private-configuration\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:10.209961 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.209813 2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-trusted-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:10.209961 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.209823 2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-ca-trust-extracted\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:10.209961 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.209833 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdqvf\" (UniqueName: \"kubernetes.io/projected/3af9fa53-b2cb-4c58-93c3-c790ed0467f6-kube-api-access-vdqvf\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:18:10.481081 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.481039 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/init-config-reloader/0.log" Sep 16 01:18:10.653021 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.652990 2567 generic.go:356] "Generic (PLEG): container finished" podID="3af9fa53-b2cb-4c58-93c3-c790ed0467f6" containerID="4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da" exitCode=0 Sep 16 01:18:10.653224 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.653045 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" Sep 16 01:18:10.653224 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.653068 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" event={"ID":"3af9fa53-b2cb-4c58-93c3-c790ed0467f6","Type":"ContainerDied","Data":"4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da"} Sep 16 01:18:10.653224 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.653108 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d8cdf7994-q99wn" event={"ID":"3af9fa53-b2cb-4c58-93c3-c790ed0467f6","Type":"ContainerDied","Data":"a6a7467a32b92d25ad3b77b680013f41c56085b74eae9be4b484d9958031cf52"} Sep 16 01:18:10.653224 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.653128 2567 scope.go:117] "RemoveContainer" containerID="4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da" Sep 16 01:18:10.662568 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.662550 2567 scope.go:117] "RemoveContainer" containerID="4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da" Sep 16 01:18:10.662823 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:18:10.662804 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da\": container with ID starting with 4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da not found: ID does not exist" containerID="4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da" Sep 16 01:18:10.662882 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.662838 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da"} err="failed to get container status \"4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da\": rpc error: code = NotFound desc = could not find container \"4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da\": container with ID starting with 4630b223ab36f5045a053d3e96ea96f3309358081b5532c36b5fbee767e7c4da not found: ID does not exist" Sep 16 01:18:10.675583 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.675550 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d8cdf7994-q99wn"] Sep 16 01:18:10.679137 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.679113 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/alertmanager/0.log" Sep 16 01:18:10.679975 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.679953 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d8cdf7994-q99wn"] Sep 16 01:18:10.881037 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:10.880952 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/config-reloader/0.log" Sep 16 01:18:11.078834 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:11.078806 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/kube-rbac-proxy-web/0.log" Sep 16 01:18:11.279156 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:11.279125 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/kube-rbac-proxy/0.log" Sep 16 01:18:11.479601 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:11.479572 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/kube-rbac-proxy-metric/0.log" Sep 16 01:18:11.679465 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:11.679390 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/prom-label-proxy/0.log" Sep 16 01:18:11.881863 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:11.881831 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-zs9pd_9ae8823d-2dd3-4b66-a92a-aaca58fad1c6/cluster-monitoring-operator/0.log" Sep 16 01:18:11.938650 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:11.938565 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af9fa53-b2cb-4c58-93c3-c790ed0467f6" path="/var/lib/kubelet/pods/3af9fa53-b2cb-4c58-93c3-c790ed0467f6/volumes" Sep 16 01:18:13.078708 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:13.078678 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2pw7_cccf8013-4766-4267-87e7-735069220a4e/init-textfile/0.log" Sep 16 01:18:13.281028 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:13.280989 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2pw7_cccf8013-4766-4267-87e7-735069220a4e/node-exporter/0.log" Sep 16 01:18:13.479506 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:13.479437 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2pw7_cccf8013-4766-4267-87e7-735069220a4e/kube-rbac-proxy/0.log" Sep 16 01:18:14.878879 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:14.878850 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zk5tk_84bc2eeb-73e5-4859-ae77-6a8543135f2c/kube-rbac-proxy-main/0.log" Sep 16 01:18:15.079197 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:15.079171 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zk5tk_84bc2eeb-73e5-4859-ae77-6a8543135f2c/kube-rbac-proxy-self/0.log" Sep 16 01:18:15.279228 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:15.279201 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zk5tk_84bc2eeb-73e5-4859-ae77-6a8543135f2c/openshift-state-metrics/0.log" Sep 16 01:18:15.478715 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:15.478679 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/init-config-reloader/0.log" Sep 16 01:18:15.681391 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:15.681303 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/prometheus/0.log" Sep 16 01:18:15.879190 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:15.879148 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/config-reloader/0.log" Sep 16 01:18:16.079073 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:16.079043 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/thanos-sidecar/0.log" Sep 16 01:18:16.279460 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:16.279430 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/kube-rbac-proxy-web/0.log" Sep 16 01:18:16.479438 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:16.479361 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/kube-rbac-proxy/0.log" Sep 16 01:18:16.678482 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:16.678447 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/kube-rbac-proxy-thanos/0.log" Sep 16 01:18:16.881985 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:16.881951 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-pjvgl_4aab2a54-aac1-4a3a-970c-2fd77e17bdeb/prometheus-operator/0.log" Sep 16 01:18:17.081612 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:17.081588 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-pjvgl_4aab2a54-aac1-4a3a-970c-2fd77e17bdeb/kube-rbac-proxy/0.log" Sep 16 01:18:19.681637 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:19.681603 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-gkl42_c28672a2-2930-4f95-80b3-0cccb3ce00d9/download-server/0.log" Sep 16 01:18:20.279305 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:20.279258 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vsrx7_b9a324b1-00bf-4a0d-9e52-bef0adfbcd32/serve-healthcheck-canary/0.log" Sep 16 01:18:49.199712 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:49.199624 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:18:49.219136 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:49.219107 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:18:49.822297 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:18:49.822247 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 01:19:06.339452 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.339409 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-577b44b7b8-89hvd"] Sep 16 01:19:06.339900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.339879 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f58ce5-574e-4879-97f8-578842c82295" containerName="console" Sep 16 01:19:06.339943 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.339905 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f58ce5-574e-4879-97f8-578842c82295" containerName="console" Sep 16 01:19:06.339943 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.339928 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af9fa53-b2cb-4c58-93c3-c790ed0467f6" containerName="registry" Sep 16 01:19:06.339943 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.339936 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af9fa53-b2cb-4c58-93c3-c790ed0467f6" containerName="registry" Sep 16 01:19:06.340097 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.340079 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af9fa53-b2cb-4c58-93c3-c790ed0467f6" containerName="registry" Sep 16 01:19:06.340149 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.340103 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f58ce5-574e-4879-97f8-578842c82295" containerName="console" Sep 16 01:19:06.343555 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.343531 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.346539 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.346512 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 16 01:19:06.346539 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.346519 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 16 01:19:06.346777 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.346741 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 16 01:19:06.346858 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.346837 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-f4scr\"" Sep 16 01:19:06.346919 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.346888 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 16 01:19:06.347811 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.347642 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 16 01:19:06.352665 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.352243 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 16 01:19:06.354043 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.353627 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-577b44b7b8-89hvd"] Sep 16 01:19:06.494556 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.494516 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7edf63ed-3a7a-4ddc-adef-27f4cd055734-serving-certs-ca-bundle\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.494556 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.494556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7edf63ed-3a7a-4ddc-adef-27f4cd055734-telemeter-trusted-ca-bundle\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.494787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.494577 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-telemeter-client-tls\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.494787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.494597 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-federate-client-tls\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.494787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.494689 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.494787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.494723 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-secret-telemeter-client\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.494787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.494753 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d7gc\" (UniqueName: \"kubernetes.io/projected/7edf63ed-3a7a-4ddc-adef-27f4cd055734-kube-api-access-5d7gc\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.494967 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.494815 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7edf63ed-3a7a-4ddc-adef-27f4cd055734-metrics-client-ca\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.595944 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.595843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.595944 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.595890 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-secret-telemeter-client\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.595944 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.595932 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d7gc\" (UniqueName: \"kubernetes.io/projected/7edf63ed-3a7a-4ddc-adef-27f4cd055734-kube-api-access-5d7gc\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.596223 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.595981 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7edf63ed-3a7a-4ddc-adef-27f4cd055734-metrics-client-ca\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.596223 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.596015 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7edf63ed-3a7a-4ddc-adef-27f4cd055734-serving-certs-ca-bundle\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.596223 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.596038 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7edf63ed-3a7a-4ddc-adef-27f4cd055734-telemeter-trusted-ca-bundle\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.596223 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.596069 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-telemeter-client-tls\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.596223 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.596099 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-federate-client-tls\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.596837 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.596803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7edf63ed-3a7a-4ddc-adef-27f4cd055734-serving-certs-ca-bundle\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.596964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.596843 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7edf63ed-3a7a-4ddc-adef-27f4cd055734-metrics-client-ca\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.597007 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.596963 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7edf63ed-3a7a-4ddc-adef-27f4cd055734-telemeter-trusted-ca-bundle\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.603952 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.599142 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-telemeter-client-tls\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.603952 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.599170 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-federate-client-tls\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.603952 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.599407 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-secret-telemeter-client\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.603952 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.599481 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7edf63ed-3a7a-4ddc-adef-27f4cd055734-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.605137 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.604961 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d7gc\" (UniqueName: \"kubernetes.io/projected/7edf63ed-3a7a-4ddc-adef-27f4cd055734-kube-api-access-5d7gc\") pod \"telemeter-client-577b44b7b8-89hvd\" (UID: \"7edf63ed-3a7a-4ddc-adef-27f4cd055734\") " pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.657227 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.657169 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" Sep 16 01:19:06.932844 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:06.932757 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-577b44b7b8-89hvd"] Sep 16 01:19:06.935591 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:19:06.935560 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7edf63ed_3a7a_4ddc_adef_27f4cd055734.slice/crio-bd412e18c56bd59cc8bc091caddeb165d40e6db0c72b8d7508076f01468d3c91 WatchSource:0}: Error finding container bd412e18c56bd59cc8bc091caddeb165d40e6db0c72b8d7508076f01468d3c91: Status 404 returned error can't find the container with id bd412e18c56bd59cc8bc091caddeb165d40e6db0c72b8d7508076f01468d3c91 Sep 16 01:19:07.878741 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:07.878701 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" event={"ID":"7edf63ed-3a7a-4ddc-adef-27f4cd055734","Type":"ContainerStarted","Data":"bd412e18c56bd59cc8bc091caddeb165d40e6db0c72b8d7508076f01468d3c91"} Sep 16 01:19:08.884632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:08.884600 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" event={"ID":"7edf63ed-3a7a-4ddc-adef-27f4cd055734","Type":"ContainerStarted","Data":"50e842625c23ff1010f2ab4352d5db8c0b39873a3ca3dec42cd425a8649e78aa"} Sep 16 01:19:08.884926 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:08.884834 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" event={"ID":"7edf63ed-3a7a-4ddc-adef-27f4cd055734","Type":"ContainerStarted","Data":"e34fe35f345e98363439ca85f692d820e4202416725bf59f35ad7104ae942a24"} Sep 16 01:19:08.884926 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:08.884866 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" event={"ID":"7edf63ed-3a7a-4ddc-adef-27f4cd055734","Type":"ContainerStarted","Data":"7bf075b8eae98985a45e74fc9638690acf09094451cd6cecdf2f2ed2a67fcf36"} Sep 16 01:19:08.905421 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:19:08.905371 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-577b44b7b8-89hvd" podStartSLOduration=1.130791053 podStartE2EDuration="2.905353926s" totalImagesPullingTime="1.774562873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:19:08.90437438 +0000 UTC m=+199.490176591" watchObservedRunningTime="2026-09-16 01:19:08.905353926 +0000 UTC m=+199.491156137" Sep 16 01:20:49.844298 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:20:49.844257 2567 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 01:21:01.285950 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.285920 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t"] Sep 16 01:21:01.292110 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.289333 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t" Sep 16 01:21:01.292346 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.292326 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-qrxl9\"" Sep 16 01:21:01.292682 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.292663 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 01:21:01.293329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.293304 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 01:21:01.296324 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.296296 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t"] Sep 16 01:21:01.456007 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.455966 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9fdp\" (UniqueName: \"kubernetes.io/projected/e3ddae19-090c-4f9d-b920-afc298c8797c-kube-api-access-q9fdp\") pod \"openshift-lws-operator-fd8ccff4c-j7h6t\" (UID: \"e3ddae19-090c-4f9d-b920-afc298c8797c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t" Sep 16 01:21:01.456007 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.456006 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3ddae19-090c-4f9d-b920-afc298c8797c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j7h6t\" (UID: \"e3ddae19-090c-4f9d-b920-afc298c8797c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t" Sep 16 01:21:01.557138 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.557044 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fdp\" (UniqueName: \"kubernetes.io/projected/e3ddae19-090c-4f9d-b920-afc298c8797c-kube-api-access-q9fdp\") pod \"openshift-lws-operator-fd8ccff4c-j7h6t\" (UID: \"e3ddae19-090c-4f9d-b920-afc298c8797c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t" Sep 16 01:21:01.557138 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.557087 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3ddae19-090c-4f9d-b920-afc298c8797c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j7h6t\" (UID: \"e3ddae19-090c-4f9d-b920-afc298c8797c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t" Sep 16 01:21:01.557459 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.557442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3ddae19-090c-4f9d-b920-afc298c8797c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j7h6t\" (UID: \"e3ddae19-090c-4f9d-b920-afc298c8797c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t" Sep 16 01:21:01.565005 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.564979 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fdp\" (UniqueName: \"kubernetes.io/projected/e3ddae19-090c-4f9d-b920-afc298c8797c-kube-api-access-q9fdp\") pod \"openshift-lws-operator-fd8ccff4c-j7h6t\" (UID: \"e3ddae19-090c-4f9d-b920-afc298c8797c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t" Sep 16 01:21:01.607827 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.607794 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t" Sep 16 01:21:01.878900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.878875 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t"] Sep 16 01:21:01.881209 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:21:01.881180 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3ddae19_090c_4f9d_b920_afc298c8797c.slice/crio-f7b5dfa2af9b0165d908bb948a92f5a8940a5438bac2daa7ef613ab604d7119c WatchSource:0}: Error finding container f7b5dfa2af9b0165d908bb948a92f5a8940a5438bac2daa7ef613ab604d7119c: Status 404 returned error can't find the container with id f7b5dfa2af9b0165d908bb948a92f5a8940a5438bac2daa7ef613ab604d7119c Sep 16 01:21:01.882816 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:01.882800 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 01:21:02.316363 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:02.316332 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t" event={"ID":"e3ddae19-090c-4f9d-b920-afc298c8797c","Type":"ContainerStarted","Data":"f7b5dfa2af9b0165d908bb948a92f5a8940a5438bac2daa7ef613ab604d7119c"} Sep 16 01:21:05.332020 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:05.331979 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t" event={"ID":"e3ddae19-090c-4f9d-b920-afc298c8797c","Type":"ContainerStarted","Data":"2ebbecd7a46dbf8fd7eab9a58b663790d7e71282d3a9f04512cf986278be7edd"} Sep 16 01:21:05.346973 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:05.346920 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j7h6t" podStartSLOduration=1.7626960710000001 podStartE2EDuration="4.346899415s" totalImagesPullingTime="2.584203344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:21:05.346203165 +0000 UTC m=+315.932005377" watchObservedRunningTime="2026-09-16 01:21:05.346899415 +0000 UTC m=+315.932701629" Sep 16 01:21:24.660910 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.660873 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd"] Sep 16 01:21:24.664325 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.664310 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.668755 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.668729 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 16 01:21:24.668890 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.668801 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 16 01:21:24.670093 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.670071 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 16 01:21:24.670203 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.670139 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v9c2d\"" Sep 16 01:21:24.680682 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.680661 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd"] Sep 16 01:21:24.748917 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.748884 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d765e931-03cd-4401-866d-497c73323a41-cert\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.749087 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.748932 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d765e931-03cd-4401-866d-497c73323a41-metrics-cert\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.749087 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.748958 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d765e931-03cd-4401-866d-497c73323a41-manager-config\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.749087 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.749034 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglnd\" (UniqueName: \"kubernetes.io/projected/d765e931-03cd-4401-866d-497c73323a41-kube-api-access-nglnd\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.800538 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.800504 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n"] Sep 16 01:21:24.803800 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.803783 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.812347 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.812320 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 01:21:24.812610 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.812591 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 01:21:24.812703 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.812621 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 01:21:24.813528 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.813507 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8s6lc\"" Sep 16 01:21:24.813635 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.813590 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 01:21:24.839945 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.839916 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n"] Sep 16 01:21:24.850250 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.850226 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe3f9fc5-8813-403c-9ab6-beb36d75145d-manifests\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.850405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.850261 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe3f9fc5-8813-403c-9ab6-beb36d75145d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.850405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.850312 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d765e931-03cd-4401-866d-497c73323a41-metrics-cert\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.850405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.850350 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe3f9fc5-8813-403c-9ab6-beb36d75145d-tmp\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.850566 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.850492 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe3f9fc5-8813-403c-9ab6-beb36d75145d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.850566 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.850536 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d765e931-03cd-4401-866d-497c73323a41-manager-config\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.850666 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.850580 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglnd\" (UniqueName: \"kubernetes.io/projected/d765e931-03cd-4401-866d-497c73323a41-kube-api-access-nglnd\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.850666 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.850617 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vfzx\" (UniqueName: \"kubernetes.io/projected/fe3f9fc5-8813-403c-9ab6-beb36d75145d-kube-api-access-6vfzx\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.850666 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.850664 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d765e931-03cd-4401-866d-497c73323a41-cert\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.851293 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.851213 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d765e931-03cd-4401-866d-497c73323a41-manager-config\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.853147 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.853124 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d765e931-03cd-4401-866d-497c73323a41-cert\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.853247 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.853213 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d765e931-03cd-4401-866d-497c73323a41-metrics-cert\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.867881 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.867849 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglnd\" (UniqueName: \"kubernetes.io/projected/d765e931-03cd-4401-866d-497c73323a41-kube-api-access-nglnd\") pod \"lws-controller-manager-65dff4dd97-jb2bd\" (UID: \"d765e931-03cd-4401-866d-497c73323a41\") " pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:24.951896 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.951813 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vfzx\" (UniqueName: \"kubernetes.io/projected/fe3f9fc5-8813-403c-9ab6-beb36d75145d-kube-api-access-6vfzx\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.951896 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.951880 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe3f9fc5-8813-403c-9ab6-beb36d75145d-manifests\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.952102 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.951909 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe3f9fc5-8813-403c-9ab6-beb36d75145d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.952102 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.951938 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe3f9fc5-8813-403c-9ab6-beb36d75145d-tmp\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.952102 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.951962 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe3f9fc5-8813-403c-9ab6-beb36d75145d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.952346 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.952320 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe3f9fc5-8813-403c-9ab6-beb36d75145d-manifests\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.954498 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.954466 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe3f9fc5-8813-403c-9ab6-beb36d75145d-tmp\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.954621 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.954541 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe3f9fc5-8813-403c-9ab6-beb36d75145d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.954684 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.954670 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe3f9fc5-8813-403c-9ab6-beb36d75145d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.959652 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.959628 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vfzx\" (UniqueName: \"kubernetes.io/projected/fe3f9fc5-8813-403c-9ab6-beb36d75145d-kube-api-access-6vfzx\") pod \"opendatahub-operator-controller-manager-5f84c66984-4889n\" (UID: \"fe3f9fc5-8813-403c-9ab6-beb36d75145d\") " pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:24.973550 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:24.973525 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:25.114388 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:25.114349 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:25.267531 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:25.267474 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd"] Sep 16 01:21:25.270972 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:21:25.270944 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd765e931_03cd_4401_866d_497c73323a41.slice/crio-7012f9c40f124f92ac35bf1f23a4d9ab79f55c90033305fc7056f69837586faa WatchSource:0}: Error finding container 7012f9c40f124f92ac35bf1f23a4d9ab79f55c90033305fc7056f69837586faa: Status 404 returned error can't find the container with id 7012f9c40f124f92ac35bf1f23a4d9ab79f55c90033305fc7056f69837586faa Sep 16 01:21:25.389970 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:25.389945 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n"] Sep 16 01:21:25.391734 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:21:25.391711 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3f9fc5_8813_403c_9ab6_beb36d75145d.slice/crio-5a71a264968b7070951f1089f0796558740b1c0c6bb3840ebea5e306557dfeb2 WatchSource:0}: Error finding container 5a71a264968b7070951f1089f0796558740b1c0c6bb3840ebea5e306557dfeb2: Status 404 returned error can't find the container with id 5a71a264968b7070951f1089f0796558740b1c0c6bb3840ebea5e306557dfeb2 Sep 16 01:21:25.411262 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:25.411233 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" event={"ID":"fe3f9fc5-8813-403c-9ab6-beb36d75145d","Type":"ContainerStarted","Data":"5a71a264968b7070951f1089f0796558740b1c0c6bb3840ebea5e306557dfeb2"} Sep 16 01:21:25.412451 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:25.412429 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" event={"ID":"d765e931-03cd-4401-866d-497c73323a41","Type":"ContainerStarted","Data":"7012f9c40f124f92ac35bf1f23a4d9ab79f55c90033305fc7056f69837586faa"} Sep 16 01:21:29.434374 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:29.434330 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" event={"ID":"d765e931-03cd-4401-866d-497c73323a41","Type":"ContainerStarted","Data":"9d3b898ff64596ef8b74884e33ea3c5e3948a66473ea84981f3cba49d4aa3ae4"} Sep 16 01:21:29.434813 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:29.434416 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:29.436013 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:29.435988 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" event={"ID":"fe3f9fc5-8813-403c-9ab6-beb36d75145d","Type":"ContainerStarted","Data":"8fd84c36df80b2babdf8b7fbde810fdef80eff01d46007446ddd0043062ab365"} Sep 16 01:21:29.455315 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:29.455243 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" podStartSLOduration=1.9955383370000002 podStartE2EDuration="5.455225546s" totalImagesPullingTime="3.459687209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:21:29.453437072 +0000 UTC m=+340.039239284" watchObservedRunningTime="2026-09-16 01:21:29.455225546 +0000 UTC m=+340.041027756" Sep 16 01:21:30.442083 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:30.442049 2567 generic.go:356] "Generic (PLEG): container finished" podID="fe3f9fc5-8813-403c-9ab6-beb36d75145d" containerID="8fd84c36df80b2babdf8b7fbde810fdef80eff01d46007446ddd0043062ab365" exitCode=0 Sep 16 01:21:30.442570 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:30.442141 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" event={"ID":"fe3f9fc5-8813-403c-9ab6-beb36d75145d","Type":"ContainerDied","Data":"8fd84c36df80b2babdf8b7fbde810fdef80eff01d46007446ddd0043062ab365"} Sep 16 01:21:31.449077 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:31.448987 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" event={"ID":"fe3f9fc5-8813-403c-9ab6-beb36d75145d","Type":"ContainerStarted","Data":"b0ab8d2b237456959d7bdb23c2bf99942efedc7ee16e17b8084dc36065dedb50"} Sep 16 01:21:31.449489 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:31.449129 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:31.477672 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:31.477621 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" podStartSLOduration=2.494213586 podStartE2EDuration="7.477602289s" totalImagesPullingTime="3.937226841s" totalInitContainerRuntime="1.046161862s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:21:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:21:31.4749275 +0000 UTC m=+342.060729712" watchObservedRunningTime="2026-09-16 01:21:31.477602289 +0000 UTC m=+342.063404569" Sep 16 01:21:40.444964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:40.444934 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65dff4dd97-jb2bd" Sep 16 01:21:42.455877 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:42.455843 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f84c66984-4889n" Sep 16 01:21:47.346480 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.346447 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-569b779d49-cpwrh"] Sep 16 01:21:47.357497 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.357452 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" Sep 16 01:21:47.360300 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.360237 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 16 01:21:47.360300 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.360264 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 01:21:47.360500 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.360366 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 16 01:21:47.360623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.360601 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-569b779d49-cpwrh"] Sep 16 01:21:47.360623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.360617 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 01:21:47.361494 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.361466 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fx69d\"" Sep 16 01:21:47.456515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.456483 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rngr\" (UniqueName: \"kubernetes.io/projected/c4323475-b3bb-457f-84fc-46b6ba632999-kube-api-access-9rngr\") pod \"kube-auth-proxy-569b779d49-cpwrh\" (UID: \"c4323475-b3bb-457f-84fc-46b6ba632999\") " pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" Sep 16 01:21:47.456515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.456524 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4323475-b3bb-457f-84fc-46b6ba632999-tls-certs\") pod \"kube-auth-proxy-569b779d49-cpwrh\" (UID: \"c4323475-b3bb-457f-84fc-46b6ba632999\") " pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" Sep 16 01:21:47.456731 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.456541 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4323475-b3bb-457f-84fc-46b6ba632999-tmp\") pod \"kube-auth-proxy-569b779d49-cpwrh\" (UID: \"c4323475-b3bb-457f-84fc-46b6ba632999\") " pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" Sep 16 01:21:47.557601 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.557520 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rngr\" (UniqueName: \"kubernetes.io/projected/c4323475-b3bb-457f-84fc-46b6ba632999-kube-api-access-9rngr\") pod \"kube-auth-proxy-569b779d49-cpwrh\" (UID: \"c4323475-b3bb-457f-84fc-46b6ba632999\") " pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" Sep 16 01:21:47.557797 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.557628 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4323475-b3bb-457f-84fc-46b6ba632999-tls-certs\") pod \"kube-auth-proxy-569b779d49-cpwrh\" (UID: \"c4323475-b3bb-457f-84fc-46b6ba632999\") " pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" Sep 16 01:21:47.557797 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.557654 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4323475-b3bb-457f-84fc-46b6ba632999-tmp\") pod \"kube-auth-proxy-569b779d49-cpwrh\" (UID: \"c4323475-b3bb-457f-84fc-46b6ba632999\") " pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" Sep 16 01:21:47.560161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.560128 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4323475-b3bb-457f-84fc-46b6ba632999-tmp\") pod \"kube-auth-proxy-569b779d49-cpwrh\" (UID: \"c4323475-b3bb-457f-84fc-46b6ba632999\") " pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" Sep 16 01:21:47.560340 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.560217 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4323475-b3bb-457f-84fc-46b6ba632999-tls-certs\") pod \"kube-auth-proxy-569b779d49-cpwrh\" (UID: \"c4323475-b3bb-457f-84fc-46b6ba632999\") " pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" Sep 16 01:21:47.565723 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.565694 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rngr\" (UniqueName: \"kubernetes.io/projected/c4323475-b3bb-457f-84fc-46b6ba632999-kube-api-access-9rngr\") pod \"kube-auth-proxy-569b779d49-cpwrh\" (UID: \"c4323475-b3bb-457f-84fc-46b6ba632999\") " pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" Sep 16 01:21:47.668176 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.668090 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" Sep 16 01:21:47.941424 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:47.941403 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-569b779d49-cpwrh"] Sep 16 01:21:47.943652 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:21:47.943624 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4323475_b3bb_457f_84fc_46b6ba632999.slice/crio-2e567ec69bcd01de780d708d8e7b8bb0143cbd8e6f3b6675d5c5f3844fe57818 WatchSource:0}: Error finding container 2e567ec69bcd01de780d708d8e7b8bb0143cbd8e6f3b6675d5c5f3844fe57818: Status 404 returned error can't find the container with id 2e567ec69bcd01de780d708d8e7b8bb0143cbd8e6f3b6675d5c5f3844fe57818 Sep 16 01:21:48.522723 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:48.522688 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" event={"ID":"c4323475-b3bb-457f-84fc-46b6ba632999","Type":"ContainerStarted","Data":"2e567ec69bcd01de780d708d8e7b8bb0143cbd8e6f3b6675d5c5f3844fe57818"} Sep 16 01:21:50.955093 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:50.955072 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 16 01:21:51.540995 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:51.540963 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" event={"ID":"c4323475-b3bb-457f-84fc-46b6ba632999","Type":"ContainerStarted","Data":"52d61f9ce1d90da82d8dd55ee61e7596f6d473df2921131c9721fa64758293b5"} Sep 16 01:21:51.555769 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:21:51.555719 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-569b779d49-cpwrh" podStartSLOduration=1.548852476 podStartE2EDuration="4.555702495s" totalImagesPullingTime="3.006850019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:21:51.55494753 +0000 UTC m=+362.140749742" watchObservedRunningTime="2026-09-16 01:21:51.555702495 +0000 UTC m=+362.141504706" Sep 16 01:23:24.762400 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.762367 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9"] Sep 16 01:23:24.765484 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.765468 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:24.767939 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.767915 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 16 01:23:24.768050 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.767915 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 16 01:23:24.768050 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.767962 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q65k2\"" Sep 16 01:23:24.768899 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.768879 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 01:23:24.768996 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.768926 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 01:23:24.773355 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.773331 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9"] Sep 16 01:23:24.898429 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.898394 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bef538a8-b86c-4f3f-872f-a64cbf5391a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-vwdb9\" (UID: \"bef538a8-b86c-4f3f-872f-a64cbf5391a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:24.898429 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.898432 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6j89\" (UniqueName: \"kubernetes.io/projected/bef538a8-b86c-4f3f-872f-a64cbf5391a9-kube-api-access-r6j89\") pod \"kuadrant-console-plugin-fbdd4f445-vwdb9\" (UID: \"bef538a8-b86c-4f3f-872f-a64cbf5391a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:24.898655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.898541 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bef538a8-b86c-4f3f-872f-a64cbf5391a9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-vwdb9\" (UID: \"bef538a8-b86c-4f3f-872f-a64cbf5391a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:24.999390 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.999359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bef538a8-b86c-4f3f-872f-a64cbf5391a9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-vwdb9\" (UID: \"bef538a8-b86c-4f3f-872f-a64cbf5391a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:24.999597 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.999438 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bef538a8-b86c-4f3f-872f-a64cbf5391a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-vwdb9\" (UID: \"bef538a8-b86c-4f3f-872f-a64cbf5391a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:24.999597 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:24.999469 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6j89\" (UniqueName: \"kubernetes.io/projected/bef538a8-b86c-4f3f-872f-a64cbf5391a9-kube-api-access-r6j89\") pod \"kuadrant-console-plugin-fbdd4f445-vwdb9\" (UID: \"bef538a8-b86c-4f3f-872f-a64cbf5391a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:24.999597 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:23:24.999577 2567 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 16 01:23:24.999758 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:23:24.999656 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bef538a8-b86c-4f3f-872f-a64cbf5391a9-plugin-serving-cert podName:bef538a8-b86c-4f3f-872f-a64cbf5391a9 nodeName:}" failed. No retries permitted until 2026-09-16 01:23:25.499637488 +0000 UTC m=+456.085439702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/bef538a8-b86c-4f3f-872f-a64cbf5391a9-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-vwdb9" (UID: "bef538a8-b86c-4f3f-872f-a64cbf5391a9") : secret "plugin-serving-cert" not found Sep 16 01:23:25.000040 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:25.000020 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bef538a8-b86c-4f3f-872f-a64cbf5391a9-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-vwdb9\" (UID: \"bef538a8-b86c-4f3f-872f-a64cbf5391a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:25.007962 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:25.007938 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6j89\" (UniqueName: \"kubernetes.io/projected/bef538a8-b86c-4f3f-872f-a64cbf5391a9-kube-api-access-r6j89\") pod \"kuadrant-console-plugin-fbdd4f445-vwdb9\" (UID: \"bef538a8-b86c-4f3f-872f-a64cbf5391a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:25.503355 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:25.503324 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bef538a8-b86c-4f3f-872f-a64cbf5391a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-vwdb9\" (UID: \"bef538a8-b86c-4f3f-872f-a64cbf5391a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:25.505926 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:25.505907 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bef538a8-b86c-4f3f-872f-a64cbf5391a9-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-vwdb9\" (UID: \"bef538a8-b86c-4f3f-872f-a64cbf5391a9\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:25.675493 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:25.675457 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" Sep 16 01:23:25.944428 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:25.944405 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9"] Sep 16 01:23:25.946637 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:23:25.946608 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbef538a8_b86c_4f3f_872f_a64cbf5391a9.slice/crio-e260702486c84bece3a8cbd566d92499f79096a8acf157e69b96e38ecee88519 WatchSource:0}: Error finding container e260702486c84bece3a8cbd566d92499f79096a8acf157e69b96e38ecee88519: Status 404 returned error can't find the container with id e260702486c84bece3a8cbd566d92499f79096a8acf157e69b96e38ecee88519 Sep 16 01:23:25.971180 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:25.971145 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" event={"ID":"bef538a8-b86c-4f3f-872f-a64cbf5391a9","Type":"ContainerStarted","Data":"e260702486c84bece3a8cbd566d92499f79096a8acf157e69b96e38ecee88519"} Sep 16 01:23:31.004534 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:31.004493 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" event={"ID":"bef538a8-b86c-4f3f-872f-a64cbf5391a9","Type":"ContainerStarted","Data":"3f60385ea3c4f7adcc7c70d51bc6b09f4b288d370687d695b301612969ce940f"} Sep 16 01:23:31.018429 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:31.018379 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-vwdb9" podStartSLOduration=2.521595755 podStartE2EDuration="7.018364437s" totalImagesPullingTime="4.496768682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:23:31.017932038 +0000 UTC m=+461.603734249" watchObservedRunningTime="2026-09-16 01:23:31.018364437 +0000 UTC m=+461.604166647" Sep 16 01:23:34.947935 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:34.947899 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p"] Sep 16 01:23:34.955260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:34.955230 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:34.957911 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:34.957891 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-h9x8n\"" Sep 16 01:23:34.966291 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:34.966248 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p"] Sep 16 01:23:35.087809 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.087769 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6286e65f-9386-48a9-9010-fcefa8657163-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-9jn4p\" (UID: \"6286e65f-9386-48a9-9010-fcefa8657163\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:35.087978 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.087878 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdtmg\" (UniqueName: \"kubernetes.io/projected/6286e65f-9386-48a9-9010-fcefa8657163-kube-api-access-wdtmg\") pod \"kuadrant-operator-controller-manager-5f685c496f-9jn4p\" (UID: \"6286e65f-9386-48a9-9010-fcefa8657163\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:35.188359 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.188325 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdtmg\" (UniqueName: \"kubernetes.io/projected/6286e65f-9386-48a9-9010-fcefa8657163-kube-api-access-wdtmg\") pod \"kuadrant-operator-controller-manager-5f685c496f-9jn4p\" (UID: \"6286e65f-9386-48a9-9010-fcefa8657163\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:35.188497 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.188406 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6286e65f-9386-48a9-9010-fcefa8657163-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-9jn4p\" (UID: \"6286e65f-9386-48a9-9010-fcefa8657163\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:35.188777 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.188757 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6286e65f-9386-48a9-9010-fcefa8657163-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-9jn4p\" (UID: \"6286e65f-9386-48a9-9010-fcefa8657163\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:35.195842 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.195813 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdtmg\" (UniqueName: \"kubernetes.io/projected/6286e65f-9386-48a9-9010-fcefa8657163-kube-api-access-wdtmg\") pod \"kuadrant-operator-controller-manager-5f685c496f-9jn4p\" (UID: \"6286e65f-9386-48a9-9010-fcefa8657163\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:35.265848 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.265817 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:35.544627 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.544551 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p"] Sep 16 01:23:35.547842 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:23:35.547811 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6286e65f_9386_48a9_9010_fcefa8657163.slice/crio-3f1eca0ccabdd46707387e4dcdddd36b7033f0c123b2dab6a0c8238c9fe0d81a WatchSource:0}: Error finding container 3f1eca0ccabdd46707387e4dcdddd36b7033f0c123b2dab6a0c8238c9fe0d81a: Status 404 returned error can't find the container with id 3f1eca0ccabdd46707387e4dcdddd36b7033f0c123b2dab6a0c8238c9fe0d81a Sep 16 01:23:35.654847 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.654805 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p"] Sep 16 01:23:35.662219 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.662188 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p"] Sep 16 01:23:35.675979 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.675945 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh"] Sep 16 01:23:35.680755 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.680731 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:23:35.692171 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.692136 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcpnb\" (UniqueName: \"kubernetes.io/projected/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-kube-api-access-tcpnb\") pod \"kuadrant-operator-controller-manager-5f685c496f-hm5zh\" (UID: \"e2e5dafa-c615-4174-a8f9-66cf7a980fc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:23:35.692333 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.692196 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-hm5zh\" (UID: \"e2e5dafa-c615-4174-a8f9-66cf7a980fc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:23:35.694245 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.694224 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh"] Sep 16 01:23:35.793042 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.793007 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcpnb\" (UniqueName: \"kubernetes.io/projected/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-kube-api-access-tcpnb\") pod \"kuadrant-operator-controller-manager-5f685c496f-hm5zh\" (UID: \"e2e5dafa-c615-4174-a8f9-66cf7a980fc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:23:35.793042 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.793045 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-hm5zh\" (UID: \"e2e5dafa-c615-4174-a8f9-66cf7a980fc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:23:35.793476 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.793459 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-hm5zh\" (UID: \"e2e5dafa-c615-4174-a8f9-66cf7a980fc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:23:35.802043 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.801973 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcpnb\" (UniqueName: \"kubernetes.io/projected/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-kube-api-access-tcpnb\") pod \"kuadrant-operator-controller-manager-5f685c496f-hm5zh\" (UID: \"e2e5dafa-c615-4174-a8f9-66cf7a980fc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:23:35.993700 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:35.993667 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:23:36.272756 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:36.272720 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh"] Sep 16 01:23:36.275890 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:23:36.275862 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e5dafa_c615_4174_a8f9_66cf7a980fc3.slice/crio-8b8e4d4985bf6fdc6ff3a7140a1fa710e99fa0e6051f5f3aee9d67a67031365c WatchSource:0}: Error finding container 8b8e4d4985bf6fdc6ff3a7140a1fa710e99fa0e6051f5f3aee9d67a67031365c: Status 404 returned error can't find the container with id 8b8e4d4985bf6fdc6ff3a7140a1fa710e99fa0e6051f5f3aee9d67a67031365c Sep 16 01:23:37.038828 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:37.038783 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" event={"ID":"e2e5dafa-c615-4174-a8f9-66cf7a980fc3","Type":"ContainerStarted","Data":"8b8e4d4985bf6fdc6ff3a7140a1fa710e99fa0e6051f5f3aee9d67a67031365c"} Sep 16 01:23:41.062942 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.062909 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" podUID="6286e65f-9386-48a9-9010-fcefa8657163" containerName="manager" containerID="cri-o://119f98df7ef83f325d1d00c791bbfb4d410592e704535c5d24ba4519529f6337" gracePeriod=2 Sep 16 01:23:41.064757 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.064731 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" event={"ID":"e2e5dafa-c615-4174-a8f9-66cf7a980fc3","Type":"ContainerStarted","Data":"285ab9b606be0f3916dfbb1474d8f71aeab7bdb75a9c7becbed3d0865c1db89b"} Sep 16 01:23:41.064874 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.064859 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:23:41.065070 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:23:41.065048 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-9jn4p\" is forbidden: User \"system:node:ip-10-0-142-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-56.ec2.internal' and this object" podUID="6286e65f-9386-48a9-9010-fcefa8657163" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:41.066851 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:23:41.066825 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-9jn4p\" is forbidden: User \"system:node:ip-10-0-142-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-56.ec2.internal' and this object" podUID="6286e65f-9386-48a9-9010-fcefa8657163" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:41.087225 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.087178 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" podStartSLOduration=2.008858125 podStartE2EDuration="6.087160347s" totalImagesPullingTime="4.078302222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:23:41.084366412 +0000 UTC m=+471.670168612" watchObservedRunningTime="2026-09-16 01:23:41.087160347 +0000 UTC m=+471.672962558" Sep 16 01:23:41.310622 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.310593 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:41.312892 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:23:41.312867 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-9jn4p\" is forbidden: User \"system:node:ip-10-0-142-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-56.ec2.internal' and this object" podUID="6286e65f-9386-48a9-9010-fcefa8657163" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:41.345159 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.345076 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdtmg\" (UniqueName: \"kubernetes.io/projected/6286e65f-9386-48a9-9010-fcefa8657163-kube-api-access-wdtmg\") pod \"6286e65f-9386-48a9-9010-fcefa8657163\" (UID: \"6286e65f-9386-48a9-9010-fcefa8657163\") " Sep 16 01:23:41.345299 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.345186 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6286e65f-9386-48a9-9010-fcefa8657163-extensions-socket-volume\") pod \"6286e65f-9386-48a9-9010-fcefa8657163\" (UID: \"6286e65f-9386-48a9-9010-fcefa8657163\") " Sep 16 01:23:41.345483 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.345461 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6286e65f-9386-48a9-9010-fcefa8657163-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6286e65f-9386-48a9-9010-fcefa8657163" (UID: "6286e65f-9386-48a9-9010-fcefa8657163"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:23:41.347593 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.347567 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6286e65f-9386-48a9-9010-fcefa8657163-kube-api-access-wdtmg" (OuterVolumeSpecName: "kube-api-access-wdtmg") pod "6286e65f-9386-48a9-9010-fcefa8657163" (UID: "6286e65f-9386-48a9-9010-fcefa8657163"). InnerVolumeSpecName "kube-api-access-wdtmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:23:41.446201 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.446160 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6286e65f-9386-48a9-9010-fcefa8657163-extensions-socket-volume\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:23:41.446201 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.446194 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdtmg\" (UniqueName: \"kubernetes.io/projected/6286e65f-9386-48a9-9010-fcefa8657163-kube-api-access-wdtmg\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:23:41.940021 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:41.939983 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6286e65f-9386-48a9-9010-fcefa8657163" path="/var/lib/kubelet/pods/6286e65f-9386-48a9-9010-fcefa8657163/volumes" Sep 16 01:23:42.071206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:42.071171 2567 generic.go:356] "Generic (PLEG): container finished" podID="6286e65f-9386-48a9-9010-fcefa8657163" containerID="119f98df7ef83f325d1d00c791bbfb4d410592e704535c5d24ba4519529f6337" exitCode=2 Sep 16 01:23:42.071646 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:42.071213 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:42.071646 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:42.071266 2567 scope.go:117] "RemoveContainer" containerID="119f98df7ef83f325d1d00c791bbfb4d410592e704535c5d24ba4519529f6337" Sep 16 01:23:42.073519 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:23:42.073493 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-9jn4p\" is forbidden: User \"system:node:ip-10-0-142-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-56.ec2.internal' and this object" podUID="6286e65f-9386-48a9-9010-fcefa8657163" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:42.075532 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:23:42.075500 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-9jn4p\" is forbidden: User \"system:node:ip-10-0-142-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-56.ec2.internal' and this object" podUID="6286e65f-9386-48a9-9010-fcefa8657163" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-9jn4p" Sep 16 01:23:42.079663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:42.079630 2567 scope.go:117] "RemoveContainer" containerID="119f98df7ef83f325d1d00c791bbfb4d410592e704535c5d24ba4519529f6337" Sep 16 01:23:42.079954 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:23:42.079936 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119f98df7ef83f325d1d00c791bbfb4d410592e704535c5d24ba4519529f6337\": container with ID starting with 119f98df7ef83f325d1d00c791bbfb4d410592e704535c5d24ba4519529f6337 not found: ID does not exist" containerID="119f98df7ef83f325d1d00c791bbfb4d410592e704535c5d24ba4519529f6337" Sep 16 01:23:42.080026 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:42.079965 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119f98df7ef83f325d1d00c791bbfb4d410592e704535c5d24ba4519529f6337"} err="failed to get container status \"119f98df7ef83f325d1d00c791bbfb4d410592e704535c5d24ba4519529f6337\": rpc error: code = NotFound desc = could not find container \"119f98df7ef83f325d1d00c791bbfb4d410592e704535c5d24ba4519529f6337\": container with ID starting with 119f98df7ef83f325d1d00c791bbfb4d410592e704535c5d24ba4519529f6337 not found: ID does not exist" Sep 16 01:23:52.074517 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:23:52.074487 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:24:04.641012 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:04.640966 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh"] Sep 16 01:24:04.641572 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:04.641167 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" podUID="e2e5dafa-c615-4174-a8f9-66cf7a980fc3" containerName="manager" containerID="cri-o://285ab9b606be0f3916dfbb1474d8f71aeab7bdb75a9c7becbed3d0865c1db89b" gracePeriod=10 Sep 16 01:24:05.176678 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:05.176638 2567 generic.go:356] "Generic (PLEG): container finished" podID="e2e5dafa-c615-4174-a8f9-66cf7a980fc3" containerID="285ab9b606be0f3916dfbb1474d8f71aeab7bdb75a9c7becbed3d0865c1db89b" exitCode=0 Sep 16 01:24:05.176845 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:05.176711 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" event={"ID":"e2e5dafa-c615-4174-a8f9-66cf7a980fc3","Type":"ContainerDied","Data":"285ab9b606be0f3916dfbb1474d8f71aeab7bdb75a9c7becbed3d0865c1db89b"} Sep 16 01:24:05.423741 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:05.423705 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:24:05.555867 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:05.555829 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-extensions-socket-volume\") pod \"e2e5dafa-c615-4174-a8f9-66cf7a980fc3\" (UID: \"e2e5dafa-c615-4174-a8f9-66cf7a980fc3\") " Sep 16 01:24:05.556074 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:05.555909 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcpnb\" (UniqueName: \"kubernetes.io/projected/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-kube-api-access-tcpnb\") pod \"e2e5dafa-c615-4174-a8f9-66cf7a980fc3\" (UID: \"e2e5dafa-c615-4174-a8f9-66cf7a980fc3\") " Sep 16 01:24:05.556226 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:05.556191 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e2e5dafa-c615-4174-a8f9-66cf7a980fc3" (UID: "e2e5dafa-c615-4174-a8f9-66cf7a980fc3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:24:05.558161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:05.558139 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-kube-api-access-tcpnb" (OuterVolumeSpecName: "kube-api-access-tcpnb") pod "e2e5dafa-c615-4174-a8f9-66cf7a980fc3" (UID: "e2e5dafa-c615-4174-a8f9-66cf7a980fc3"). InnerVolumeSpecName "kube-api-access-tcpnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:24:05.657369 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:05.657330 2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-extensions-socket-volume\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:24:05.657369 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:05.657359 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcpnb\" (UniqueName: \"kubernetes.io/projected/e2e5dafa-c615-4174-a8f9-66cf7a980fc3-kube-api-access-tcpnb\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:24:06.184891 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:06.184859 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" Sep 16 01:24:06.184891 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:06.184872 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh" event={"ID":"e2e5dafa-c615-4174-a8f9-66cf7a980fc3","Type":"ContainerDied","Data":"8b8e4d4985bf6fdc6ff3a7140a1fa710e99fa0e6051f5f3aee9d67a67031365c"} Sep 16 01:24:06.185119 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:06.184942 2567 scope.go:117] "RemoveContainer" containerID="285ab9b606be0f3916dfbb1474d8f71aeab7bdb75a9c7becbed3d0865c1db89b" Sep 16 01:24:06.202415 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:06.202348 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh"] Sep 16 01:24:06.206002 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:06.205974 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hm5zh"] Sep 16 01:24:07.939216 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:07.939183 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2e5dafa-c615-4174-a8f9-66cf7a980fc3" path="/var/lib/kubelet/pods/e2e5dafa-c615-4174-a8f9-66cf7a980fc3/volumes" Sep 16 01:24:26.623796 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.623760 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tbqt5"] Sep 16 01:24:26.624161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.624098 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e5dafa-c615-4174-a8f9-66cf7a980fc3" containerName="manager" Sep 16 01:24:26.624161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.624109 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e5dafa-c615-4174-a8f9-66cf7a980fc3" containerName="manager" Sep 16 01:24:26.624161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.624123 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6286e65f-9386-48a9-9010-fcefa8657163" containerName="manager" Sep 16 01:24:26.624161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.624128 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6286e65f-9386-48a9-9010-fcefa8657163" containerName="manager" Sep 16 01:24:26.624304 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.624185 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6286e65f-9386-48a9-9010-fcefa8657163" containerName="manager" Sep 16 01:24:26.624304 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.624195 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e5dafa-c615-4174-a8f9-66cf7a980fc3" containerName="manager" Sep 16 01:24:26.626787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.626771 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:26.629143 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.629120 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 16 01:24:26.635559 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.635535 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tbqt5"] Sep 16 01:24:26.655042 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.654999 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/837ec867-bf1b-43c6-b73a-fb4e12837553-config-file\") pod \"limitador-limitador-8588749977-tbqt5\" (UID: \"837ec867-bf1b-43c6-b73a-fb4e12837553\") " pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:26.655182 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.655059 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2lpk\" (UniqueName: \"kubernetes.io/projected/837ec867-bf1b-43c6-b73a-fb4e12837553-kube-api-access-r2lpk\") pod \"limitador-limitador-8588749977-tbqt5\" (UID: \"837ec867-bf1b-43c6-b73a-fb4e12837553\") " pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:26.713720 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.713688 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tbqt5"] Sep 16 01:24:26.756469 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.756437 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lpk\" (UniqueName: \"kubernetes.io/projected/837ec867-bf1b-43c6-b73a-fb4e12837553-kube-api-access-r2lpk\") pod \"limitador-limitador-8588749977-tbqt5\" (UID: \"837ec867-bf1b-43c6-b73a-fb4e12837553\") " pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:26.756629 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.756539 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/837ec867-bf1b-43c6-b73a-fb4e12837553-config-file\") pod \"limitador-limitador-8588749977-tbqt5\" (UID: \"837ec867-bf1b-43c6-b73a-fb4e12837553\") " pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:26.757171 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.757151 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/837ec867-bf1b-43c6-b73a-fb4e12837553-config-file\") pod \"limitador-limitador-8588749977-tbqt5\" (UID: \"837ec867-bf1b-43c6-b73a-fb4e12837553\") " pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:26.763799 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.763774 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lpk\" (UniqueName: \"kubernetes.io/projected/837ec867-bf1b-43c6-b73a-fb4e12837553-kube-api-access-r2lpk\") pod \"limitador-limitador-8588749977-tbqt5\" (UID: \"837ec867-bf1b-43c6-b73a-fb4e12837553\") " pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:26.938161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:26.938073 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:27.205548 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:27.205522 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tbqt5"] Sep 16 01:24:27.207485 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:24:27.207460 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837ec867_bf1b_43c6_b73a_fb4e12837553.slice/crio-ce2045537ebf329b4a8de94e38151714e4bb89accd795270240c2ca84bfb4cfc WatchSource:0}: Error finding container ce2045537ebf329b4a8de94e38151714e4bb89accd795270240c2ca84bfb4cfc: Status 404 returned error can't find the container with id ce2045537ebf329b4a8de94e38151714e4bb89accd795270240c2ca84bfb4cfc Sep 16 01:24:27.279252 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:27.279219 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" event={"ID":"837ec867-bf1b-43c6-b73a-fb4e12837553","Type":"ContainerStarted","Data":"ce2045537ebf329b4a8de94e38151714e4bb89accd795270240c2ca84bfb4cfc"} Sep 16 01:24:27.374860 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:27.374832 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zlnzj"] Sep 16 01:24:27.377603 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:27.377587 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" Sep 16 01:24:27.380264 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:27.380243 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-g98ls\"" Sep 16 01:24:27.385083 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:27.385058 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zlnzj"] Sep 16 01:24:27.461804 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:27.461720 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vtl\" (UniqueName: \"kubernetes.io/projected/d7c5321f-e983-4545-afe8-c4f632fe5766-kube-api-access-n2vtl\") pod \"authorino-7bcfd8f9cd-zlnzj\" (UID: \"d7c5321f-e983-4545-afe8-c4f632fe5766\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" Sep 16 01:24:27.563073 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:27.563030 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vtl\" (UniqueName: \"kubernetes.io/projected/d7c5321f-e983-4545-afe8-c4f632fe5766-kube-api-access-n2vtl\") pod \"authorino-7bcfd8f9cd-zlnzj\" (UID: \"d7c5321f-e983-4545-afe8-c4f632fe5766\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" Sep 16 01:24:27.570513 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:27.570485 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vtl\" (UniqueName: \"kubernetes.io/projected/d7c5321f-e983-4545-afe8-c4f632fe5766-kube-api-access-n2vtl\") pod \"authorino-7bcfd8f9cd-zlnzj\" (UID: \"d7c5321f-e983-4545-afe8-c4f632fe5766\") " pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" Sep 16 01:24:27.687388 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:27.687355 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" Sep 16 01:24:27.954263 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:27.954220 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zlnzj"] Sep 16 01:24:27.957164 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:24:27.957138 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7c5321f_e983_4545_afe8_c4f632fe5766.slice/crio-32c54ebe219e0ea50c0261f3527c0a1c2a6605b5f3740428f0ab750a222334ce WatchSource:0}: Error finding container 32c54ebe219e0ea50c0261f3527c0a1c2a6605b5f3740428f0ab750a222334ce: Status 404 returned error can't find the container with id 32c54ebe219e0ea50c0261f3527c0a1c2a6605b5f3740428f0ab750a222334ce Sep 16 01:24:28.287829 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:28.287793 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" event={"ID":"d7c5321f-e983-4545-afe8-c4f632fe5766","Type":"ContainerStarted","Data":"32c54ebe219e0ea50c0261f3527c0a1c2a6605b5f3740428f0ab750a222334ce"} Sep 16 01:24:29.294724 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:29.294692 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" event={"ID":"837ec867-bf1b-43c6-b73a-fb4e12837553","Type":"ContainerStarted","Data":"c7b349c622df8be8bde5cc006fb20a242a7fb1ff4ec85b5a41975ce514530997"} Sep 16 01:24:29.295079 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:29.294771 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:29.309752 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:29.309706 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" podStartSLOduration=1.909506225 podStartE2EDuration="3.3096913s" totalImagesPullingTime="1.400185075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:24:29.308557123 +0000 UTC m=+519.894359334" watchObservedRunningTime="2026-09-16 01:24:29.3096913 +0000 UTC m=+519.895493510" Sep 16 01:24:30.302639 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:30.302455 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" event={"ID":"d7c5321f-e983-4545-afe8-c4f632fe5766","Type":"ContainerStarted","Data":"d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f"} Sep 16 01:24:30.318471 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:30.318422 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" podStartSLOduration=1.099285729 podStartE2EDuration="3.318404748s" totalImagesPullingTime="2.219119019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:24:30.31652614 +0000 UTC m=+520.902328360" watchObservedRunningTime="2026-09-16 01:24:30.318404748 +0000 UTC m=+520.904206959" Sep 16 01:24:30.348546 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:30.348510 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zlnzj"] Sep 16 01:24:32.312043 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:32.312004 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" podUID="d7c5321f-e983-4545-afe8-c4f632fe5766" containerName="authorino" containerID="cri-o://d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f" gracePeriod=30 Sep 16 01:24:32.554901 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:32.554877 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" Sep 16 01:24:32.608846 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:32.608756 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2vtl\" (UniqueName: \"kubernetes.io/projected/d7c5321f-e983-4545-afe8-c4f632fe5766-kube-api-access-n2vtl\") pod \"d7c5321f-e983-4545-afe8-c4f632fe5766\" (UID: \"d7c5321f-e983-4545-afe8-c4f632fe5766\") " Sep 16 01:24:32.610919 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:32.610892 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c5321f-e983-4545-afe8-c4f632fe5766-kube-api-access-n2vtl" (OuterVolumeSpecName: "kube-api-access-n2vtl") pod "d7c5321f-e983-4545-afe8-c4f632fe5766" (UID: "d7c5321f-e983-4545-afe8-c4f632fe5766"). InnerVolumeSpecName "kube-api-access-n2vtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:24:32.709532 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:32.709503 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vtl\" (UniqueName: \"kubernetes.io/projected/d7c5321f-e983-4545-afe8-c4f632fe5766-kube-api-access-n2vtl\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:24:33.318009 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:33.317974 2567 generic.go:356] "Generic (PLEG): container finished" podID="d7c5321f-e983-4545-afe8-c4f632fe5766" containerID="d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f" exitCode=0 Sep 16 01:24:33.318410 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:33.318022 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" Sep 16 01:24:33.318410 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:33.318061 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" event={"ID":"d7c5321f-e983-4545-afe8-c4f632fe5766","Type":"ContainerDied","Data":"d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f"} Sep 16 01:24:33.318410 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:33.318109 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-zlnzj" event={"ID":"d7c5321f-e983-4545-afe8-c4f632fe5766","Type":"ContainerDied","Data":"32c54ebe219e0ea50c0261f3527c0a1c2a6605b5f3740428f0ab750a222334ce"} Sep 16 01:24:33.318410 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:33.318130 2567 scope.go:117] "RemoveContainer" containerID="d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f" Sep 16 01:24:33.327219 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:33.327203 2567 scope.go:117] "RemoveContainer" containerID="d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f" Sep 16 01:24:33.327526 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:24:33.327506 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f\": container with ID starting with d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f not found: ID does not exist" containerID="d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f" Sep 16 01:24:33.327590 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:33.327534 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f"} err="failed to get container status \"d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f\": rpc error: code = NotFound desc = could not find container \"d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f\": container with ID starting with d1875ce12feddacc57c26afad33234effef65e904d31792e2b291fd5a5be1a8f not found: ID does not exist" Sep 16 01:24:33.338447 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:33.338418 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zlnzj"] Sep 16 01:24:33.343324 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:33.343303 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-zlnzj"] Sep 16 01:24:33.938725 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:33.938690 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7c5321f-e983-4545-afe8-c4f632fe5766" path="/var/lib/kubelet/pods/d7c5321f-e983-4545-afe8-c4f632fe5766/volumes" Sep 16 01:24:39.826367 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:39.826335 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tbqt5"] Sep 16 01:24:39.826900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:39.826528 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" podUID="837ec867-bf1b-43c6-b73a-fb4e12837553" containerName="limitador" containerID="cri-o://c7b349c622df8be8bde5cc006fb20a242a7fb1ff4ec85b5a41975ce514530997" gracePeriod=30 Sep 16 01:24:39.827293 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:39.827259 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:40.354750 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:40.354716 2567 generic.go:356] "Generic (PLEG): container finished" podID="837ec867-bf1b-43c6-b73a-fb4e12837553" containerID="c7b349c622df8be8bde5cc006fb20a242a7fb1ff4ec85b5a41975ce514530997" exitCode=0 Sep 16 01:24:40.354943 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:40.354783 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" event={"ID":"837ec867-bf1b-43c6-b73a-fb4e12837553","Type":"ContainerDied","Data":"c7b349c622df8be8bde5cc006fb20a242a7fb1ff4ec85b5a41975ce514530997"} Sep 16 01:24:40.373994 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:40.373970 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:40.475241 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:40.475164 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2lpk\" (UniqueName: \"kubernetes.io/projected/837ec867-bf1b-43c6-b73a-fb4e12837553-kube-api-access-r2lpk\") pod \"837ec867-bf1b-43c6-b73a-fb4e12837553\" (UID: \"837ec867-bf1b-43c6-b73a-fb4e12837553\") " Sep 16 01:24:40.475241 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:40.475218 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/837ec867-bf1b-43c6-b73a-fb4e12837553-config-file\") pod \"837ec867-bf1b-43c6-b73a-fb4e12837553\" (UID: \"837ec867-bf1b-43c6-b73a-fb4e12837553\") " Sep 16 01:24:40.475605 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:40.475586 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837ec867-bf1b-43c6-b73a-fb4e12837553-config-file" (OuterVolumeSpecName: "config-file") pod "837ec867-bf1b-43c6-b73a-fb4e12837553" (UID: "837ec867-bf1b-43c6-b73a-fb4e12837553"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:24:40.477472 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:40.477447 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837ec867-bf1b-43c6-b73a-fb4e12837553-kube-api-access-r2lpk" (OuterVolumeSpecName: "kube-api-access-r2lpk") pod "837ec867-bf1b-43c6-b73a-fb4e12837553" (UID: "837ec867-bf1b-43c6-b73a-fb4e12837553"). InnerVolumeSpecName "kube-api-access-r2lpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:24:40.576521 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:40.576482 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2lpk\" (UniqueName: \"kubernetes.io/projected/837ec867-bf1b-43c6-b73a-fb4e12837553-kube-api-access-r2lpk\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:24:40.576521 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:40.576515 2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/837ec867-bf1b-43c6-b73a-fb4e12837553-config-file\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:24:41.360476 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:41.360445 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" event={"ID":"837ec867-bf1b-43c6-b73a-fb4e12837553","Type":"ContainerDied","Data":"ce2045537ebf329b4a8de94e38151714e4bb89accd795270240c2ca84bfb4cfc"} Sep 16 01:24:41.360476 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:41.360472 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-tbqt5" Sep 16 01:24:41.360960 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:41.360505 2567 scope.go:117] "RemoveContainer" containerID="c7b349c622df8be8bde5cc006fb20a242a7fb1ff4ec85b5a41975ce514530997" Sep 16 01:24:41.381623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:41.381596 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tbqt5"] Sep 16 01:24:41.385141 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:41.385116 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-tbqt5"] Sep 16 01:24:41.938924 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:41.938877 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837ec867-bf1b-43c6-b73a-fb4e12837553" path="/var/lib/kubelet/pods/837ec867-bf1b-43c6-b73a-fb4e12837553/volumes" Sep 16 01:24:42.717526 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.717491 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk"] Sep 16 01:24:42.718035 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.718015 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837ec867-bf1b-43c6-b73a-fb4e12837553" containerName="limitador" Sep 16 01:24:42.718091 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.718041 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="837ec867-bf1b-43c6-b73a-fb4e12837553" containerName="limitador" Sep 16 01:24:42.718091 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.718057 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c5321f-e983-4545-afe8-c4f632fe5766" containerName="authorino" Sep 16 01:24:42.718091 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.718065 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c5321f-e983-4545-afe8-c4f632fe5766" containerName="authorino" Sep 16 01:24:42.718180 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.718172 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="837ec867-bf1b-43c6-b73a-fb4e12837553" containerName="limitador" Sep 16 01:24:42.718213 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.718188 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c5321f-e983-4545-afe8-c4f632fe5766" containerName="authorino" Sep 16 01:24:42.720609 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.720588 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" Sep 16 01:24:42.723262 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.723240 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 01:24:42.723405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.723317 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 16 01:24:42.724377 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.724357 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4kzkh\"" Sep 16 01:24:42.724491 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.724358 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 01:24:42.727767 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.727744 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk"] Sep 16 01:24:42.796918 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.796879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/73baf411-6c15-4374-8781-f665cf3c286a-data\") pod \"postgres-69fb4594cf-n7rzk\" (UID: \"73baf411-6c15-4374-8781-f665cf3c286a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" Sep 16 01:24:42.797108 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.797020 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlczg\" (UniqueName: \"kubernetes.io/projected/73baf411-6c15-4374-8781-f665cf3c286a-kube-api-access-qlczg\") pod \"postgres-69fb4594cf-n7rzk\" (UID: \"73baf411-6c15-4374-8781-f665cf3c286a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" Sep 16 01:24:42.898232 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.898198 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/73baf411-6c15-4374-8781-f665cf3c286a-data\") pod \"postgres-69fb4594cf-n7rzk\" (UID: \"73baf411-6c15-4374-8781-f665cf3c286a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" Sep 16 01:24:42.898413 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.898323 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlczg\" (UniqueName: \"kubernetes.io/projected/73baf411-6c15-4374-8781-f665cf3c286a-kube-api-access-qlczg\") pod \"postgres-69fb4594cf-n7rzk\" (UID: \"73baf411-6c15-4374-8781-f665cf3c286a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" Sep 16 01:24:42.898597 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.898574 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/73baf411-6c15-4374-8781-f665cf3c286a-data\") pod \"postgres-69fb4594cf-n7rzk\" (UID: \"73baf411-6c15-4374-8781-f665cf3c286a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" Sep 16 01:24:42.906917 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:42.906893 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlczg\" (UniqueName: \"kubernetes.io/projected/73baf411-6c15-4374-8781-f665cf3c286a-kube-api-access-qlczg\") pod \"postgres-69fb4594cf-n7rzk\" (UID: \"73baf411-6c15-4374-8781-f665cf3c286a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" Sep 16 01:24:43.032475 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:43.032433 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" Sep 16 01:24:43.298887 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:43.298807 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk"] Sep 16 01:24:43.301884 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:24:43.301858 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73baf411_6c15_4374_8781_f665cf3c286a.slice/crio-fe42a9c192da7566ca83afa6df675f479210398afb73d5e2feac78eb1ad8a186 WatchSource:0}: Error finding container fe42a9c192da7566ca83afa6df675f479210398afb73d5e2feac78eb1ad8a186: Status 404 returned error can't find the container with id fe42a9c192da7566ca83afa6df675f479210398afb73d5e2feac78eb1ad8a186 Sep 16 01:24:43.371098 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:43.371062 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" event={"ID":"73baf411-6c15-4374-8781-f665cf3c286a","Type":"ContainerStarted","Data":"fe42a9c192da7566ca83afa6df675f479210398afb73d5e2feac78eb1ad8a186"} Sep 16 01:24:46.386640 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:46.386602 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" event={"ID":"73baf411-6c15-4374-8781-f665cf3c286a","Type":"ContainerStarted","Data":"4b735e552818ed47481206aff2cf1f0f0445df320c60dfa68d1b8bd050d3a3f6"} Sep 16 01:24:46.387163 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:46.386662 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" Sep 16 01:24:46.400270 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:46.400223 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" podStartSLOduration=1.881017749 podStartE2EDuration="4.40020448s" totalImagesPullingTime="2.519186731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:24:46.400067356 +0000 UTC m=+536.985869567" watchObservedRunningTime="2026-09-16 01:24:46.40020448 +0000 UTC m=+536.986006693" Sep 16 01:24:52.421689 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:52.421658 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-n7rzk" Sep 16 01:24:52.968935 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:52.968900 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-g4ksw"] Sep 16 01:24:52.974620 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:52.974602 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-g4ksw" Sep 16 01:24:52.977089 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:52.977063 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-g98ls\"" Sep 16 01:24:52.979187 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:52.979161 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-g4ksw"] Sep 16 01:24:53.086588 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:53.086555 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49tdv\" (UniqueName: \"kubernetes.io/projected/892b5e18-cff6-4457-a1c9-5d438b291caf-kube-api-access-49tdv\") pod \"authorino-557f876f69-g4ksw\" (UID: \"892b5e18-cff6-4457-a1c9-5d438b291caf\") " pod="kuadrant-system/authorino-557f876f69-g4ksw" Sep 16 01:24:53.145849 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:53.145812 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-g4ksw"] Sep 16 01:24:53.146100 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:24:53.146080 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-49tdv], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-g4ksw" podUID="892b5e18-cff6-4457-a1c9-5d438b291caf" Sep 16 01:24:53.187040 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:53.187003 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tdv\" (UniqueName: \"kubernetes.io/projected/892b5e18-cff6-4457-a1c9-5d438b291caf-kube-api-access-49tdv\") pod \"authorino-557f876f69-g4ksw\" (UID: \"892b5e18-cff6-4457-a1c9-5d438b291caf\") " pod="kuadrant-system/authorino-557f876f69-g4ksw" Sep 16 01:24:53.198660 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:53.198627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49tdv\" (UniqueName: \"kubernetes.io/projected/892b5e18-cff6-4457-a1c9-5d438b291caf-kube-api-access-49tdv\") pod \"authorino-557f876f69-g4ksw\" (UID: \"892b5e18-cff6-4457-a1c9-5d438b291caf\") " pod="kuadrant-system/authorino-557f876f69-g4ksw" Sep 16 01:24:53.418107 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:53.418074 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-g4ksw" Sep 16 01:24:53.431332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:53.431306 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-g4ksw" Sep 16 01:24:53.591126 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:53.591079 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49tdv\" (UniqueName: \"kubernetes.io/projected/892b5e18-cff6-4457-a1c9-5d438b291caf-kube-api-access-49tdv\") pod \"892b5e18-cff6-4457-a1c9-5d438b291caf\" (UID: \"892b5e18-cff6-4457-a1c9-5d438b291caf\") " Sep 16 01:24:53.593340 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:53.593313 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892b5e18-cff6-4457-a1c9-5d438b291caf-kube-api-access-49tdv" (OuterVolumeSpecName: "kube-api-access-49tdv") pod "892b5e18-cff6-4457-a1c9-5d438b291caf" (UID: "892b5e18-cff6-4457-a1c9-5d438b291caf"). InnerVolumeSpecName "kube-api-access-49tdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:24:53.691964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:53.691884 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49tdv\" (UniqueName: \"kubernetes.io/projected/892b5e18-cff6-4457-a1c9-5d438b291caf-kube-api-access-49tdv\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:24:54.422564 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:54.422534 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-g4ksw" Sep 16 01:24:54.451036 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:54.450998 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-g4ksw"] Sep 16 01:24:54.455217 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:54.455186 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-g4ksw"] Sep 16 01:24:55.939212 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:24:55.939178 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="892b5e18-cff6-4457-a1c9-5d438b291caf" path="/var/lib/kubelet/pods/892b5e18-cff6-4457-a1c9-5d438b291caf/volumes" Sep 16 01:25:00.191918 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.191885 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-kl965"] Sep 16 01:25:00.196249 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.196224 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:00.198633 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.198611 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 01:25:00.198752 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.198611 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 01:25:00.198858 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.198843 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 01:25:00.198925 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.198911 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kw9lw\"" Sep 16 01:25:00.204836 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.204807 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-kl965"] Sep 16 01:25:00.251054 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.251012 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-metrics-certs\") pod \"maas-controller-75dc6cbdf-kl965\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:00.251232 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.251082 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lth4c\" (UniqueName: \"kubernetes.io/projected/52a726eb-107c-4a36-a463-abc057984b53-kube-api-access-lth4c\") pod \"maas-controller-75dc6cbdf-kl965\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:00.251232 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.251105 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-webhook-cert\") pod \"maas-controller-75dc6cbdf-kl965\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:00.352236 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.352195 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-metrics-certs\") pod \"maas-controller-75dc6cbdf-kl965\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:00.352455 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.352256 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lth4c\" (UniqueName: \"kubernetes.io/projected/52a726eb-107c-4a36-a463-abc057984b53-kube-api-access-lth4c\") pod \"maas-controller-75dc6cbdf-kl965\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:00.352455 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.352314 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-webhook-cert\") pod \"maas-controller-75dc6cbdf-kl965\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:00.352455 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:25:00.352394 2567 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 16 01:25:00.352626 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:25:00.352466 2567 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 16 01:25:00.352626 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:25:00.352500 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-metrics-certs podName:52a726eb-107c-4a36-a463-abc057984b53 nodeName:}" failed. No retries permitted until 2026-09-16 01:25:00.85247678 +0000 UTC m=+551.438278970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-metrics-certs") pod "maas-controller-75dc6cbdf-kl965" (UID: "52a726eb-107c-4a36-a463-abc057984b53") : secret "maas-controller-metrics-tls" not found Sep 16 01:25:00.352626 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:25:00.352527 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-webhook-cert podName:52a726eb-107c-4a36-a463-abc057984b53 nodeName:}" failed. No retries permitted until 2026-09-16 01:25:00.852509261 +0000 UTC m=+551.438311454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-webhook-cert") pod "maas-controller-75dc6cbdf-kl965" (UID: "52a726eb-107c-4a36-a463-abc057984b53") : secret "maas-controller-webhook-cert" not found Sep 16 01:25:00.353527 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.353504 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6859d7476b-cj9hq"] Sep 16 01:25:00.357690 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.357669 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:00.361171 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.361140 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lth4c\" (UniqueName: \"kubernetes.io/projected/52a726eb-107c-4a36-a463-abc057984b53-kube-api-access-lth4c\") pod \"maas-controller-75dc6cbdf-kl965\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:00.365970 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.365946 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6859d7476b-cj9hq"] Sep 16 01:25:00.452798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.452707 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-webhook-cert\") pod \"maas-controller-6859d7476b-cj9hq\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:00.452798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.452766 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-metrics-certs\") pod \"maas-controller-6859d7476b-cj9hq\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:00.452798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.452799 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2zs8\" (UniqueName: \"kubernetes.io/projected/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-kube-api-access-c2zs8\") pod \"maas-controller-6859d7476b-cj9hq\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:00.553366 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.553311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-webhook-cert\") pod \"maas-controller-6859d7476b-cj9hq\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:00.553557 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.553418 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-metrics-certs\") pod \"maas-controller-6859d7476b-cj9hq\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:00.553557 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.553445 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2zs8\" (UniqueName: \"kubernetes.io/projected/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-kube-api-access-c2zs8\") pod \"maas-controller-6859d7476b-cj9hq\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:00.553636 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:25:00.553566 2567 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 16 01:25:00.553636 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:25:00.553633 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-metrics-certs podName:9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b nodeName:}" failed. No retries permitted until 2026-09-16 01:25:01.05361635 +0000 UTC m=+551.639418542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-metrics-certs") pod "maas-controller-6859d7476b-cj9hq" (UID: "9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b") : secret "maas-controller-metrics-tls" not found Sep 16 01:25:00.555929 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.555906 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-webhook-cert\") pod \"maas-controller-6859d7476b-cj9hq\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:00.561591 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.561572 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2zs8\" (UniqueName: \"kubernetes.io/projected/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-kube-api-access-c2zs8\") pod \"maas-controller-6859d7476b-cj9hq\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:00.855096 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.855052 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-webhook-cert\") pod \"maas-controller-75dc6cbdf-kl965\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:00.855342 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.855186 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-metrics-certs\") pod \"maas-controller-75dc6cbdf-kl965\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:00.857641 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.857612 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-webhook-cert\") pod \"maas-controller-75dc6cbdf-kl965\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:00.857750 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:00.857722 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-metrics-certs\") pod \"maas-controller-75dc6cbdf-kl965\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:01.057056 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:01.057017 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-metrics-certs\") pod \"maas-controller-6859d7476b-cj9hq\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:01.059571 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:01.059545 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-metrics-certs\") pod \"maas-controller-6859d7476b-cj9hq\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:01.110065 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:01.109988 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:01.278753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:01.278717 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:01.380046 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:01.380018 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-kl965"] Sep 16 01:25:01.382071 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:25:01.382046 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a726eb_107c_4a36_a463_abc057984b53.slice/crio-7e99a2069098cc78d5a370b27525cc23cce59f66414b12f48f4195539fca146b WatchSource:0}: Error finding container 7e99a2069098cc78d5a370b27525cc23cce59f66414b12f48f4195539fca146b: Status 404 returned error can't find the container with id 7e99a2069098cc78d5a370b27525cc23cce59f66414b12f48f4195539fca146b Sep 16 01:25:01.458209 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:01.458175 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-kl965" event={"ID":"52a726eb-107c-4a36-a463-abc057984b53","Type":"ContainerStarted","Data":"7e99a2069098cc78d5a370b27525cc23cce59f66414b12f48f4195539fca146b"} Sep 16 01:25:01.549522 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:01.549492 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6859d7476b-cj9hq"] Sep 16 01:25:01.553434 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:25:01.553407 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9420e6b0_1d2c_4c59_ad5b_0e29cd7ee93b.slice/crio-249707f2226d9386a5130ea106dd7843f31b2b0c39139cc31455b748f37b8a6f WatchSource:0}: Error finding container 249707f2226d9386a5130ea106dd7843f31b2b0c39139cc31455b748f37b8a6f: Status 404 returned error can't find the container with id 249707f2226d9386a5130ea106dd7843f31b2b0c39139cc31455b748f37b8a6f Sep 16 01:25:02.468738 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:02.468699 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-cj9hq" event={"ID":"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b","Type":"ContainerStarted","Data":"249707f2226d9386a5130ea106dd7843f31b2b0c39139cc31455b748f37b8a6f"} Sep 16 01:25:04.480380 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:04.480318 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-kl965" event={"ID":"52a726eb-107c-4a36-a463-abc057984b53","Type":"ContainerStarted","Data":"a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357"} Sep 16 01:25:04.480839 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:04.480409 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:04.482363 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:04.482332 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-cj9hq" event={"ID":"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b","Type":"ContainerStarted","Data":"68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3"} Sep 16 01:25:04.482522 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:04.482390 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:04.500141 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:04.500090 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75dc6cbdf-kl965" podStartSLOduration=1.844510808 podStartE2EDuration="4.500076847s" totalImagesPullingTime="2.655566039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:25:04.498169619 +0000 UTC m=+555.083971832" watchObservedRunningTime="2026-09-16 01:25:04.500076847 +0000 UTC m=+555.085879057" Sep 16 01:25:04.529673 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:04.529604 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6859d7476b-cj9hq" podStartSLOduration=2.038854274 podStartE2EDuration="4.529580427s" totalImagesPullingTime="2.490726153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:25:04.527464074 +0000 UTC m=+555.113266284" watchObservedRunningTime="2026-09-16 01:25:04.529580427 +0000 UTC m=+555.115382641" Sep 16 01:25:15.492356 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.492326 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:25:15.492828 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.492395 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:15.550536 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.550500 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-kl965"] Sep 16 01:25:15.550725 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.550697 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75dc6cbdf-kl965" podUID="52a726eb-107c-4a36-a463-abc057984b53" containerName="manager" containerID="cri-o://a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357" gracePeriod=30 Sep 16 01:25:15.806402 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.806367 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:15.890450 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.890419 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-webhook-cert\") pod \"52a726eb-107c-4a36-a463-abc057984b53\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " Sep 16 01:25:15.890628 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.890523 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lth4c\" (UniqueName: \"kubernetes.io/projected/52a726eb-107c-4a36-a463-abc057984b53-kube-api-access-lth4c\") pod \"52a726eb-107c-4a36-a463-abc057984b53\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " Sep 16 01:25:15.890628 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.890553 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-metrics-certs\") pod \"52a726eb-107c-4a36-a463-abc057984b53\" (UID: \"52a726eb-107c-4a36-a463-abc057984b53\") " Sep 16 01:25:15.892946 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.892904 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "52a726eb-107c-4a36-a463-abc057984b53" (UID: "52a726eb-107c-4a36-a463-abc057984b53"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:25:15.892946 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.892926 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52a726eb-107c-4a36-a463-abc057984b53" (UID: "52a726eb-107c-4a36-a463-abc057984b53"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:25:15.893105 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.893005 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a726eb-107c-4a36-a463-abc057984b53-kube-api-access-lth4c" (OuterVolumeSpecName: "kube-api-access-lth4c") pod "52a726eb-107c-4a36-a463-abc057984b53" (UID: "52a726eb-107c-4a36-a463-abc057984b53"). InnerVolumeSpecName "kube-api-access-lth4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:25:15.991851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.991812 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lth4c\" (UniqueName: \"kubernetes.io/projected/52a726eb-107c-4a36-a463-abc057984b53-kube-api-access-lth4c\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:25:15.991851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.991844 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:25:15.991851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:15.991854 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52a726eb-107c-4a36-a463-abc057984b53-webhook-cert\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:25:16.540204 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:16.540167 2567 generic.go:356] "Generic (PLEG): container finished" podID="52a726eb-107c-4a36-a463-abc057984b53" containerID="a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357" exitCode=0 Sep 16 01:25:16.540651 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:16.540221 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-kl965" Sep 16 01:25:16.540651 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:16.540229 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-kl965" event={"ID":"52a726eb-107c-4a36-a463-abc057984b53","Type":"ContainerDied","Data":"a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357"} Sep 16 01:25:16.540651 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:16.540309 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-kl965" event={"ID":"52a726eb-107c-4a36-a463-abc057984b53","Type":"ContainerDied","Data":"7e99a2069098cc78d5a370b27525cc23cce59f66414b12f48f4195539fca146b"} Sep 16 01:25:16.540651 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:16.540332 2567 scope.go:117] "RemoveContainer" containerID="a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357" Sep 16 01:25:16.549094 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:16.549074 2567 scope.go:117] "RemoveContainer" containerID="a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357" Sep 16 01:25:16.549403 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:25:16.549382 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357\": container with ID starting with a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357 not found: ID does not exist" containerID="a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357" Sep 16 01:25:16.549475 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:16.549410 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357"} err="failed to get container status \"a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357\": rpc error: code = NotFound desc = could not find container \"a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357\": container with ID starting with a6550cd58eca8d0463cce535574101eb712ee1f2259e12bfdc12eaf1a520a357 not found: ID does not exist" Sep 16 01:25:16.555477 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:16.555455 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-kl965"] Sep 16 01:25:16.559134 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:16.559111 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-kl965"] Sep 16 01:25:17.938653 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:17.938610 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52a726eb-107c-4a36-a463-abc057984b53" path="/var/lib/kubelet/pods/52a726eb-107c-4a36-a463-abc057984b53/volumes" Sep 16 01:25:37.416837 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.416802 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm"] Sep 16 01:25:37.417299 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.417205 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52a726eb-107c-4a36-a463-abc057984b53" containerName="manager" Sep 16 01:25:37.417299 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.417217 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="52a726eb-107c-4a36-a463-abc057984b53" containerName="manager" Sep 16 01:25:37.417415 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.417315 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="52a726eb-107c-4a36-a463-abc057984b53" containerName="manager" Sep 16 01:25:37.421905 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.421882 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.424560 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.424532 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 16 01:25:37.424674 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.424635 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5nlvz\"" Sep 16 01:25:37.424674 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.424659 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 16 01:25:37.430511 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.430428 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm"] Sep 16 01:25:37.577099 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.577047 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7bb\" (UniqueName: \"kubernetes.io/projected/78058c86-f69c-450e-a194-e1d831f3cf4e-kube-api-access-xz7bb\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.577332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.577118 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78058c86-f69c-450e-a194-e1d831f3cf4e-metrics-certs\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.577332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.577186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78058c86-f69c-450e-a194-e1d831f3cf4e-maas-api-tls\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.577332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.577285 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78058c86-f69c-450e-a194-e1d831f3cf4e-openshift-service-ca\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.677898 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.677799 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7bb\" (UniqueName: \"kubernetes.io/projected/78058c86-f69c-450e-a194-e1d831f3cf4e-kube-api-access-xz7bb\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.677898 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.677860 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78058c86-f69c-450e-a194-e1d831f3cf4e-metrics-certs\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.678127 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.677921 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78058c86-f69c-450e-a194-e1d831f3cf4e-maas-api-tls\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.678127 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.677981 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78058c86-f69c-450e-a194-e1d831f3cf4e-openshift-service-ca\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.678127 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:25:37.678080 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Sep 16 01:25:37.678264 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:25:37.678164 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78058c86-f69c-450e-a194-e1d831f3cf4e-maas-api-tls podName:78058c86-f69c-450e-a194-e1d831f3cf4e nodeName:}" failed. No retries permitted until 2026-09-16 01:25:38.178143431 +0000 UTC m=+588.763945624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78058c86-f69c-450e-a194-e1d831f3cf4e-maas-api-tls") pod "maas-api-59f6566bbc-nj8bm" (UID: "78058c86-f69c-450e-a194-e1d831f3cf4e") : secret "maas-api-serving-cert" not found Sep 16 01:25:37.678719 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.678700 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78058c86-f69c-450e-a194-e1d831f3cf4e-openshift-service-ca\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.680687 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.680662 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78058c86-f69c-450e-a194-e1d831f3cf4e-metrics-certs\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.694771 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.694741 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7bb\" (UniqueName: \"kubernetes.io/projected/78058c86-f69c-450e-a194-e1d831f3cf4e-kube-api-access-xz7bb\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:37.987640 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.987608 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xzt6s"] Sep 16 01:25:37.989970 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.989938 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:25:37.993641 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.993614 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kkbcm\"" Sep 16 01:25:37.993758 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:37.993627 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 16 01:25:38.009658 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.009626 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xzt6s"] Sep 16 01:25:38.082513 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.082482 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qvxs\" (UniqueName: \"kubernetes.io/projected/968786d4-edd2-4cbb-aaf5-0af6c90372a0-kube-api-access-8qvxs\") pod \"payload-processing-5b54b7fbb7-xzt6s\" (UID: \"968786d4-edd2-4cbb-aaf5-0af6c90372a0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:25:38.082711 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.082556 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968786d4-edd2-4cbb-aaf5-0af6c90372a0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xzt6s\" (UID: \"968786d4-edd2-4cbb-aaf5-0af6c90372a0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:25:38.183158 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.183122 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qvxs\" (UniqueName: \"kubernetes.io/projected/968786d4-edd2-4cbb-aaf5-0af6c90372a0-kube-api-access-8qvxs\") pod \"payload-processing-5b54b7fbb7-xzt6s\" (UID: \"968786d4-edd2-4cbb-aaf5-0af6c90372a0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:25:38.183375 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.183189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968786d4-edd2-4cbb-aaf5-0af6c90372a0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xzt6s\" (UID: \"968786d4-edd2-4cbb-aaf5-0af6c90372a0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:25:38.183375 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.183256 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78058c86-f69c-450e-a194-e1d831f3cf4e-maas-api-tls\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:38.183922 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.183894 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968786d4-edd2-4cbb-aaf5-0af6c90372a0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xzt6s\" (UID: \"968786d4-edd2-4cbb-aaf5-0af6c90372a0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:25:38.185745 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.185720 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78058c86-f69c-450e-a194-e1d831f3cf4e-maas-api-tls\") pod \"maas-api-59f6566bbc-nj8bm\" (UID: \"78058c86-f69c-450e-a194-e1d831f3cf4e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:38.191230 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.191201 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qvxs\" (UniqueName: \"kubernetes.io/projected/968786d4-edd2-4cbb-aaf5-0af6c90372a0-kube-api-access-8qvxs\") pod \"payload-processing-5b54b7fbb7-xzt6s\" (UID: \"968786d4-edd2-4cbb-aaf5-0af6c90372a0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:25:38.302000 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.301916 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:25:38.333978 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.333944 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:38.572172 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.572148 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xzt6s"] Sep 16 01:25:38.574688 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:25:38.574657 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968786d4_edd2_4cbb_aaf5_0af6c90372a0.slice/crio-48f2ec27fe3dcea6744353b5b5f9aeae32114d3ae12535cf39cc2184a16c3349 WatchSource:0}: Error finding container 48f2ec27fe3dcea6744353b5b5f9aeae32114d3ae12535cf39cc2184a16c3349: Status 404 returned error can't find the container with id 48f2ec27fe3dcea6744353b5b5f9aeae32114d3ae12535cf39cc2184a16c3349 Sep 16 01:25:38.612492 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.612458 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm"] Sep 16 01:25:38.615839 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:25:38.615808 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78058c86_f69c_450e_a194_e1d831f3cf4e.slice/crio-d6d626777bbd543f7a5d49e060df2d56f0273d9c581817182dd3c2b365050297 WatchSource:0}: Error finding container d6d626777bbd543f7a5d49e060df2d56f0273d9c581817182dd3c2b365050297: Status 404 returned error can't find the container with id d6d626777bbd543f7a5d49e060df2d56f0273d9c581817182dd3c2b365050297 Sep 16 01:25:38.643073 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.643031 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" event={"ID":"968786d4-edd2-4cbb-aaf5-0af6c90372a0","Type":"ContainerStarted","Data":"48f2ec27fe3dcea6744353b5b5f9aeae32114d3ae12535cf39cc2184a16c3349"} Sep 16 01:25:38.644327 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:38.644300 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" event={"ID":"78058c86-f69c-450e-a194-e1d831f3cf4e","Type":"ContainerStarted","Data":"d6d626777bbd543f7a5d49e060df2d56f0273d9c581817182dd3c2b365050297"} Sep 16 01:25:41.661448 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:41.661405 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" event={"ID":"78058c86-f69c-450e-a194-e1d831f3cf4e","Type":"ContainerStarted","Data":"201cd35820ff34f38d9cd1dcdcfcfdb067aacdb95d4cc8f14c0552547a5b2a6a"} Sep 16 01:25:41.661923 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:41.661535 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:41.663095 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:41.663067 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" event={"ID":"968786d4-edd2-4cbb-aaf5-0af6c90372a0","Type":"ContainerStarted","Data":"c54417c34c660fe2647a5f9efd4b54087028982bbc2d6770e7ac920cd48bf4cb"} Sep 16 01:25:41.663239 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:41.663112 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:25:41.678885 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:41.678670 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" podStartSLOduration=2.09701241 podStartE2EDuration="4.678651239s" totalImagesPullingTime="2.581638829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:25:41.677686892 +0000 UTC m=+592.263489104" watchObservedRunningTime="2026-09-16 01:25:41.678651239 +0000 UTC m=+592.264453455" Sep 16 01:25:41.693922 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:41.693872 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" podStartSLOduration=2.0260418 podStartE2EDuration="4.693845844s" totalImagesPullingTime="2.667804044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:25:41.692025137 +0000 UTC m=+592.277827371" watchObservedRunningTime="2026-09-16 01:25:41.693845844 +0000 UTC m=+592.279648056" Sep 16 01:25:47.673547 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:47.673518 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-nj8bm" Sep 16 01:25:52.669987 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:25:52.669953 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:26:29.304665 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.304626 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc"] Sep 16 01:26:29.311570 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.308371 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.315699 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.315084 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 16 01:26:29.315699 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.315157 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 16 01:26:29.315699 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.315380 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lbkhj\"" Sep 16 01:26:29.315699 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.315491 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 16 01:26:29.318244 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.318202 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc"] Sep 16 01:26:29.446327 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.446258 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.446327 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.446333 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.446566 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.446355 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d13c24d9-f59a-4467-a255-ca862a3ec1a0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.446566 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.446377 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mc2p\" (UniqueName: \"kubernetes.io/projected/d13c24d9-f59a-4467-a255-ca862a3ec1a0-kube-api-access-2mc2p\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.446566 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.446454 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.446566 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.446499 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.547250 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.547191 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.547250 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.547251 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.547250 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.547267 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d13c24d9-f59a-4467-a255-ca862a3ec1a0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.547582 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.547297 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mc2p\" (UniqueName: \"kubernetes.io/projected/d13c24d9-f59a-4467-a255-ca862a3ec1a0-kube-api-access-2mc2p\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.547582 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.547327 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.547582 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.547350 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.547737 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.547696 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.547787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.547750 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.547851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.547787 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.549828 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.549800 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d13c24d9-f59a-4467-a255-ca862a3ec1a0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.549962 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.549944 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d13c24d9-f59a-4467-a255-ca862a3ec1a0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.555744 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.555693 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mc2p\" (UniqueName: \"kubernetes.io/projected/d13c24d9-f59a-4467-a255-ca862a3ec1a0-kube-api-access-2mc2p\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc\" (UID: \"d13c24d9-f59a-4467-a255-ca862a3ec1a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.625911 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.625872 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:29.894496 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.894473 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc"] Sep 16 01:26:29.900014 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:29.899990 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 01:26:30.891745 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:30.891715 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" event={"ID":"d13c24d9-f59a-4467-a255-ca862a3ec1a0","Type":"ContainerStarted","Data":"3a3d5a2bb34aa79e53cdb3b745ddc322db9172877a1969d69cdfd7004e4d43da"} Sep 16 01:26:31.897500 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:31.897467 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" event={"ID":"d13c24d9-f59a-4467-a255-ca862a3ec1a0","Type":"ContainerStarted","Data":"55b4fd3cda8a021819d8dea4d7478d230133a43b56028915499990059b2ad74e"} Sep 16 01:26:31.913644 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:31.913593 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" podStartSLOduration=1.947211703 podStartE2EDuration="2.913578779s" totalImagesPullingTime="966.367076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:26:31.912034023 +0000 UTC m=+642.497836233" watchObservedRunningTime="2026-09-16 01:26:31.913578779 +0000 UTC m=+642.499380989" Sep 16 01:26:37.101394 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.101355 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt"] Sep 16 01:26:37.104268 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.104244 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.106938 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.106914 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 16 01:26:37.115042 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.115019 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt"] Sep 16 01:26:37.118245 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.118221 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.118396 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.118377 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.118453 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.118432 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.118496 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.118455 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pf6s\" (UniqueName: \"kubernetes.io/projected/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-kube-api-access-4pf6s\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.118552 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.118532 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.118596 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.118564 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.219821 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.219783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.219985 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.219843 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.219985 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.219874 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pf6s\" (UniqueName: \"kubernetes.io/projected/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-kube-api-access-4pf6s\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.219985 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.219925 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.220114 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.220056 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.220155 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.220122 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.220358 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.220336 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.220469 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.220355 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.220538 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.220514 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.222213 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.222190 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.222631 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.222582 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.227653 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.227627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pf6s\" (UniqueName: \"kubernetes.io/projected/2394e6c6-0aa8-48b2-a319-536b8ff1a9ee-kube-api-access-4pf6s\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2khpt\" (UID: \"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.415676 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.415587 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:37.683722 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.683654 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt"] Sep 16 01:26:37.686048 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:26:37.686006 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2394e6c6_0aa8_48b2_a319_536b8ff1a9ee.slice/crio-b8065c4fc1c723a1628d9323542c02c8b96496701c732c4715704e3fc6b3d987 WatchSource:0}: Error finding container b8065c4fc1c723a1628d9323542c02c8b96496701c732c4715704e3fc6b3d987: Status 404 returned error can't find the container with id b8065c4fc1c723a1628d9323542c02c8b96496701c732c4715704e3fc6b3d987 Sep 16 01:26:37.927136 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.927104 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" event={"ID":"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee","Type":"ContainerStarted","Data":"e32aa89b1fce7f12f7a90c802981da357da767b0c3fccd66575693cadbfd9c5c"} Sep 16 01:26:37.927429 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.927154 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" event={"ID":"2394e6c6-0aa8-48b2-a319-536b8ff1a9ee","Type":"ContainerStarted","Data":"b8065c4fc1c723a1628d9323542c02c8b96496701c732c4715704e3fc6b3d987"} Sep 16 01:26:37.943710 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:37.943616 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" podStartSLOduration=0.772154258 podStartE2EDuration="943.599195ms" totalImagesPullingTime="171.444937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:26:37.941518708 +0000 UTC m=+648.527320918" watchObservedRunningTime="2026-09-16 01:26:37.943599195 +0000 UTC m=+648.529401406" Sep 16 01:26:39.626962 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:39.626921 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:39.627384 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:39.627027 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:39.639714 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:39.639686 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:39.948505 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:39.948430 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8b5bc" Sep 16 01:26:47.416536 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:47.416496 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:47.416947 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:47.416573 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:47.429110 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:47.429079 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:26:47.985750 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:26:47.985719 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2khpt" Sep 16 01:28:08.934187 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:08.934110 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bf58fdd4b-qqrcm"] Sep 16 01:28:08.937385 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:08.937368 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf58fdd4b-qqrcm" Sep 16 01:28:08.940847 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:08.940823 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-g98ls\"" Sep 16 01:28:08.940985 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:08.940858 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 16 01:28:08.945137 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:08.944659 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bf58fdd4b-qqrcm"] Sep 16 01:28:09.042015 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:09.041971 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnc48\" (UniqueName: \"kubernetes.io/projected/f329c94a-0925-4520-842f-0752ab16ffcf-kube-api-access-fnc48\") pod \"authorino-6bf58fdd4b-qqrcm\" (UID: \"f329c94a-0925-4520-842f-0752ab16ffcf\") " pod="kuadrant-system/authorino-6bf58fdd4b-qqrcm" Sep 16 01:28:09.042015 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:09.042007 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f329c94a-0925-4520-842f-0752ab16ffcf-tls-cert\") pod \"authorino-6bf58fdd4b-qqrcm\" (UID: \"f329c94a-0925-4520-842f-0752ab16ffcf\") " pod="kuadrant-system/authorino-6bf58fdd4b-qqrcm" Sep 16 01:28:09.142898 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:09.142866 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnc48\" (UniqueName: \"kubernetes.io/projected/f329c94a-0925-4520-842f-0752ab16ffcf-kube-api-access-fnc48\") pod \"authorino-6bf58fdd4b-qqrcm\" (UID: \"f329c94a-0925-4520-842f-0752ab16ffcf\") " pod="kuadrant-system/authorino-6bf58fdd4b-qqrcm" Sep 16 01:28:09.143057 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:09.142904 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f329c94a-0925-4520-842f-0752ab16ffcf-tls-cert\") pod \"authorino-6bf58fdd4b-qqrcm\" (UID: \"f329c94a-0925-4520-842f-0752ab16ffcf\") " pod="kuadrant-system/authorino-6bf58fdd4b-qqrcm" Sep 16 01:28:09.145503 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:09.145484 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f329c94a-0925-4520-842f-0752ab16ffcf-tls-cert\") pod \"authorino-6bf58fdd4b-qqrcm\" (UID: \"f329c94a-0925-4520-842f-0752ab16ffcf\") " pod="kuadrant-system/authorino-6bf58fdd4b-qqrcm" Sep 16 01:28:09.150352 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:09.150330 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnc48\" (UniqueName: \"kubernetes.io/projected/f329c94a-0925-4520-842f-0752ab16ffcf-kube-api-access-fnc48\") pod \"authorino-6bf58fdd4b-qqrcm\" (UID: \"f329c94a-0925-4520-842f-0752ab16ffcf\") " pod="kuadrant-system/authorino-6bf58fdd4b-qqrcm" Sep 16 01:28:09.248184 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:09.248157 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bf58fdd4b-qqrcm" Sep 16 01:28:09.516456 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:09.516425 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bf58fdd4b-qqrcm"] Sep 16 01:28:09.518913 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:28:09.518891 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf329c94a_0925_4520_842f_0752ab16ffcf.slice/crio-4cbf63e4c745d776abef4985ebd35c9167209420c15c51ba8e5a0981a336a625 WatchSource:0}: Error finding container 4cbf63e4c745d776abef4985ebd35c9167209420c15c51ba8e5a0981a336a625: Status 404 returned error can't find the container with id 4cbf63e4c745d776abef4985ebd35c9167209420c15c51ba8e5a0981a336a625 Sep 16 01:28:10.358407 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:10.358226 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bf58fdd4b-qqrcm" event={"ID":"f329c94a-0925-4520-842f-0752ab16ffcf","Type":"ContainerStarted","Data":"8bc585c6851852e5586e3d244da5809febec2238347c61c1dd3881803c2be48b"} Sep 16 01:28:10.358407 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:10.358311 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bf58fdd4b-qqrcm" event={"ID":"f329c94a-0925-4520-842f-0752ab16ffcf","Type":"ContainerStarted","Data":"4cbf63e4c745d776abef4985ebd35c9167209420c15c51ba8e5a0981a336a625"} Sep 16 01:28:10.374665 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:10.374615 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6bf58fdd4b-qqrcm" podStartSLOduration=1.8489352879999998 podStartE2EDuration="2.37459713s" totalImagesPullingTime="525.661842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:28:10.372518225 +0000 UTC m=+740.958320448" watchObservedRunningTime="2026-09-16 01:28:10.37459713 +0000 UTC m=+740.960399340" Sep 16 01:28:10.979659 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:10.979627 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6859d7476b-cj9hq"] Sep 16 01:28:10.979904 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:10.979861 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6859d7476b-cj9hq" podUID="9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b" containerName="manager" containerID="cri-o://68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3" gracePeriod=30 Sep 16 01:28:11.216679 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.216635 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xzt6s"] Sep 16 01:28:11.216923 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.216869 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" podUID="968786d4-edd2-4cbb-aaf5-0af6c90372a0" containerName="payload-processing" containerID="cri-o://c54417c34c660fe2647a5f9efd4b54087028982bbc2d6770e7ac920cd48bf4cb" gracePeriod=30 Sep 16 01:28:11.311544 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.311505 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:28:11.363178 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.363082 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2zs8\" (UniqueName: \"kubernetes.io/projected/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-kube-api-access-c2zs8\") pod \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " Sep 16 01:28:11.363664 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.363210 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-metrics-certs\") pod \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " Sep 16 01:28:11.363664 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.363271 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-webhook-cert\") pod \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\" (UID: \"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b\") " Sep 16 01:28:11.370999 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.370957 2567 generic.go:356] "Generic (PLEG): container finished" podID="968786d4-edd2-4cbb-aaf5-0af6c90372a0" containerID="c54417c34c660fe2647a5f9efd4b54087028982bbc2d6770e7ac920cd48bf4cb" exitCode=0 Sep 16 01:28:11.371182 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.371095 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" event={"ID":"968786d4-edd2-4cbb-aaf5-0af6c90372a0","Type":"ContainerDied","Data":"c54417c34c660fe2647a5f9efd4b54087028982bbc2d6770e7ac920cd48bf4cb"} Sep 16 01:28:11.372386 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.372354 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b" (UID: "9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:28:11.372911 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.372881 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b" (UID: "9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:28:11.373200 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.373172 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-kube-api-access-c2zs8" (OuterVolumeSpecName: "kube-api-access-c2zs8") pod "9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b" (UID: "9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b"). InnerVolumeSpecName "kube-api-access-c2zs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:28:11.373524 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.373436 2567 generic.go:356] "Generic (PLEG): container finished" podID="9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b" containerID="68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3" exitCode=0 Sep 16 01:28:11.373524 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.373499 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-cj9hq" Sep 16 01:28:11.373663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.373510 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-cj9hq" event={"ID":"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b","Type":"ContainerDied","Data":"68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3"} Sep 16 01:28:11.373663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.373569 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-cj9hq" event={"ID":"9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b","Type":"ContainerDied","Data":"249707f2226d9386a5130ea106dd7843f31b2b0c39139cc31455b748f37b8a6f"} Sep 16 01:28:11.373663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.373599 2567 scope.go:117] "RemoveContainer" containerID="68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3" Sep 16 01:28:11.393707 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.393682 2567 scope.go:117] "RemoveContainer" containerID="68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3" Sep 16 01:28:11.394636 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:28:11.394605 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3\": container with ID starting with 68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3 not found: ID does not exist" containerID="68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3" Sep 16 01:28:11.394729 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.394647 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3"} err="failed to get container status \"68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3\": rpc error: code = NotFound desc = could not find container \"68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3\": container with ID starting with 68e18f6bff6b3b6103a34162ede1b9ac993a2954474963e8c261ba6136bc5be3 not found: ID does not exist" Sep 16 01:28:11.404195 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.404161 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6859d7476b-cj9hq"] Sep 16 01:28:11.408611 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.408556 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6859d7476b-cj9hq"] Sep 16 01:28:11.464255 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.464231 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-webhook-cert\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:28:11.464394 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.464261 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2zs8\" (UniqueName: \"kubernetes.io/projected/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-kube-api-access-c2zs8\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:28:11.464394 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.464372 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:28:11.481738 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.481713 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:28:11.565445 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.565404 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968786d4-edd2-4cbb-aaf5-0af6c90372a0-plugins-config-volume\") pod \"968786d4-edd2-4cbb-aaf5-0af6c90372a0\" (UID: \"968786d4-edd2-4cbb-aaf5-0af6c90372a0\") " Sep 16 01:28:11.565619 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.565488 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qvxs\" (UniqueName: \"kubernetes.io/projected/968786d4-edd2-4cbb-aaf5-0af6c90372a0-kube-api-access-8qvxs\") pod \"968786d4-edd2-4cbb-aaf5-0af6c90372a0\" (UID: \"968786d4-edd2-4cbb-aaf5-0af6c90372a0\") " Sep 16 01:28:11.565754 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.565727 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968786d4-edd2-4cbb-aaf5-0af6c90372a0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "968786d4-edd2-4cbb-aaf5-0af6c90372a0" (UID: "968786d4-edd2-4cbb-aaf5-0af6c90372a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:28:11.567904 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.567864 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968786d4-edd2-4cbb-aaf5-0af6c90372a0-kube-api-access-8qvxs" (OuterVolumeSpecName: "kube-api-access-8qvxs") pod "968786d4-edd2-4cbb-aaf5-0af6c90372a0" (UID: "968786d4-edd2-4cbb-aaf5-0af6c90372a0"). InnerVolumeSpecName "kube-api-access-8qvxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:28:11.667215 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.667174 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qvxs\" (UniqueName: \"kubernetes.io/projected/968786d4-edd2-4cbb-aaf5-0af6c90372a0-kube-api-access-8qvxs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:28:11.667215 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.667206 2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968786d4-edd2-4cbb-aaf5-0af6c90372a0-plugins-config-volume\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:28:11.938892 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:11.938816 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b" path="/var/lib/kubelet/pods/9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b/volumes" Sep 16 01:28:12.092350 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.092314 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6859d7476b-n4bnr"] Sep 16 01:28:12.092707 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.092695 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="968786d4-edd2-4cbb-aaf5-0af6c90372a0" containerName="payload-processing" Sep 16 01:28:12.092752 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.092710 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="968786d4-edd2-4cbb-aaf5-0af6c90372a0" containerName="payload-processing" Sep 16 01:28:12.092752 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.092725 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b" containerName="manager" Sep 16 01:28:12.092752 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.092731 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b" containerName="manager" Sep 16 01:28:12.092851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.092795 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9420e6b0-1d2c-4c59-ad5b-0e29cd7ee93b" containerName="manager" Sep 16 01:28:12.092851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.092806 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="968786d4-edd2-4cbb-aaf5-0af6c90372a0" containerName="payload-processing" Sep 16 01:28:12.095848 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.095829 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:12.099689 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.099650 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kw9lw\"" Sep 16 01:28:12.099831 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.099692 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 01:28:12.099831 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.099692 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 01:28:12.099831 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.099650 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 01:28:12.105461 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.105434 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6859d7476b-n4bnr"] Sep 16 01:28:12.171885 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.171851 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lz27\" (UniqueName: \"kubernetes.io/projected/36927d41-75a7-4d07-b192-bdaf5294c32f-kube-api-access-6lz27\") pod \"maas-controller-6859d7476b-n4bnr\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:12.172046 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.171907 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-webhook-cert\") pod \"maas-controller-6859d7476b-n4bnr\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:12.172046 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.171932 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-metrics-certs\") pod \"maas-controller-6859d7476b-n4bnr\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:12.273076 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.273042 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lz27\" (UniqueName: \"kubernetes.io/projected/36927d41-75a7-4d07-b192-bdaf5294c32f-kube-api-access-6lz27\") pod \"maas-controller-6859d7476b-n4bnr\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:12.273214 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.273090 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-webhook-cert\") pod \"maas-controller-6859d7476b-n4bnr\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:12.273214 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.273113 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-metrics-certs\") pod \"maas-controller-6859d7476b-n4bnr\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:12.275726 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.275706 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-webhook-cert\") pod \"maas-controller-6859d7476b-n4bnr\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:12.275794 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.275750 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-metrics-certs\") pod \"maas-controller-6859d7476b-n4bnr\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:12.280383 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.280361 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lz27\" (UniqueName: \"kubernetes.io/projected/36927d41-75a7-4d07-b192-bdaf5294c32f-kube-api-access-6lz27\") pod \"maas-controller-6859d7476b-n4bnr\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:12.379931 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.379896 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" Sep 16 01:28:12.379931 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.379924 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xzt6s" event={"ID":"968786d4-edd2-4cbb-aaf5-0af6c90372a0","Type":"ContainerDied","Data":"48f2ec27fe3dcea6744353b5b5f9aeae32114d3ae12535cf39cc2184a16c3349"} Sep 16 01:28:12.380533 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.380003 2567 scope.go:117] "RemoveContainer" containerID="c54417c34c660fe2647a5f9efd4b54087028982bbc2d6770e7ac920cd48bf4cb" Sep 16 01:28:12.395656 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.395626 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xzt6s"] Sep 16 01:28:12.398898 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.398869 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xzt6s"] Sep 16 01:28:12.406938 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.406911 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:12.680945 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:12.680923 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6859d7476b-n4bnr"] Sep 16 01:28:12.682815 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:28:12.682787 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36927d41_75a7_4d07_b192_bdaf5294c32f.slice/crio-b99d2246dfc0d0c8ee1ffec663ec357f4f6921f8225491461f84d29222eda19c WatchSource:0}: Error finding container b99d2246dfc0d0c8ee1ffec663ec357f4f6921f8225491461f84d29222eda19c: Status 404 returned error can't find the container with id b99d2246dfc0d0c8ee1ffec663ec357f4f6921f8225491461f84d29222eda19c Sep 16 01:28:13.394990 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:13.394891 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-n4bnr" event={"ID":"36927d41-75a7-4d07-b192-bdaf5294c32f","Type":"ContainerStarted","Data":"aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542"} Sep 16 01:28:13.394990 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:13.394978 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-n4bnr" event={"ID":"36927d41-75a7-4d07-b192-bdaf5294c32f","Type":"ContainerStarted","Data":"b99d2246dfc0d0c8ee1ffec663ec357f4f6921f8225491461f84d29222eda19c"} Sep 16 01:28:13.395456 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:13.395124 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:28:13.411362 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:13.411312 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6859d7476b-n4bnr" podStartSLOduration=0.952012268 podStartE2EDuration="1.41129438s" totalImagesPullingTime="459.282112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:28:13.410230955 +0000 UTC m=+743.996033179" watchObservedRunningTime="2026-09-16 01:28:13.41129438 +0000 UTC m=+743.997096582" Sep 16 01:28:13.939711 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:13.939662 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="968786d4-edd2-4cbb-aaf5-0af6c90372a0" path="/var/lib/kubelet/pods/968786d4-edd2-4cbb-aaf5-0af6c90372a0/volumes" Sep 16 01:28:24.402556 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:28:24.402524 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:29:20.744010 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.743969 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt"] Sep 16 01:29:20.748071 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.748048 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.750540 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.750511 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-482177-serving-cert\"" Sep 16 01:29:20.760385 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.760359 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt"] Sep 16 01:29:20.879801 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.879763 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqzrn\" (UniqueName: \"kubernetes.io/projected/89900062-c475-4e1e-b926-49068af56754-kube-api-access-qqzrn\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.880087 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.879812 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-maas-api-tls\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.880087 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.879893 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-metrics-certs\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.880087 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.879963 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89900062-c475-4e1e-b926-49068af56754-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.981379 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.981346 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89900062-c475-4e1e-b926-49068af56754-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.981581 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.981407 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqzrn\" (UniqueName: \"kubernetes.io/projected/89900062-c475-4e1e-b926-49068af56754-kube-api-access-qqzrn\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.981581 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.981432 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-maas-api-tls\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.981581 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.981472 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-metrics-certs\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.982018 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.981995 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89900062-c475-4e1e-b926-49068af56754-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.984190 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.984152 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-metrics-certs\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.984190 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.984189 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-maas-api-tls\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:20.989628 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:20.989603 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqzrn\" (UniqueName: \"kubernetes.io/projected/89900062-c475-4e1e-b926-49068af56754-kube-api-access-qqzrn\") pod \"maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:21.062359 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:21.062263 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:21.335708 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:21.335682 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt"] Sep 16 01:29:21.337778 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:29:21.337748 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89900062_c475_4e1e_b926_49068af56754.slice/crio-34da46488978ec9a7749af03cb2e0114caecef1feeb3993c2cc2efa3360ad606 WatchSource:0}: Error finding container 34da46488978ec9a7749af03cb2e0114caecef1feeb3993c2cc2efa3360ad606: Status 404 returned error can't find the container with id 34da46488978ec9a7749af03cb2e0114caecef1feeb3993c2cc2efa3360ad606 Sep 16 01:29:21.725515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:21.725442 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" event={"ID":"89900062-c475-4e1e-b926-49068af56754","Type":"ContainerStarted","Data":"34da46488978ec9a7749af03cb2e0114caecef1feeb3993c2cc2efa3360ad606"} Sep 16 01:29:22.056795 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.056764 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr"] Sep 16 01:29:22.060338 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.060320 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.062679 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.062653 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-d0c9f2-serving-cert\"" Sep 16 01:29:22.070439 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.070017 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr"] Sep 16 01:29:22.194131 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.194087 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.194349 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.194139 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-metrics-certs\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.194349 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.194171 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxhfl\" (UniqueName: \"kubernetes.io/projected/a9471daf-6d5c-4dff-92fb-6d8b215cda34-kube-api-access-bxhfl\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.194349 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.194228 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9471daf-6d5c-4dff-92fb-6d8b215cda34-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.295196 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.295158 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.295196 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.295200 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-metrics-certs\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.295458 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.295223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxhfl\" (UniqueName: \"kubernetes.io/projected/a9471daf-6d5c-4dff-92fb-6d8b215cda34-kube-api-access-bxhfl\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.295458 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.295300 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9471daf-6d5c-4dff-92fb-6d8b215cda34-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.296020 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.295991 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9471daf-6d5c-4dff-92fb-6d8b215cda34-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.297825 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.297797 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.297920 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.297835 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-metrics-certs\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.302637 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.302614 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxhfl\" (UniqueName: \"kubernetes.io/projected/a9471daf-6d5c-4dff-92fb-6d8b215cda34-kube-api-access-bxhfl\") pod \"maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.373894 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.373806 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:22.652120 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.652056 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr"] Sep 16 01:29:22.654836 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:29:22.654803 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9471daf_6d5c_4dff_92fb_6d8b215cda34.slice/crio-42bf8d81c80ff369fdfde1e80d7f739568d12858dd036dfc4293d3b32b0017a0 WatchSource:0}: Error finding container 42bf8d81c80ff369fdfde1e80d7f739568d12858dd036dfc4293d3b32b0017a0: Status 404 returned error can't find the container with id 42bf8d81c80ff369fdfde1e80d7f739568d12858dd036dfc4293d3b32b0017a0 Sep 16 01:29:22.731850 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.731814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" event={"ID":"89900062-c475-4e1e-b926-49068af56754","Type":"ContainerStarted","Data":"addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41"} Sep 16 01:29:22.732055 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.731880 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:22.733157 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.733133 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" event={"ID":"a9471daf-6d5c-4dff-92fb-6d8b215cda34","Type":"ContainerStarted","Data":"42bf8d81c80ff369fdfde1e80d7f739568d12858dd036dfc4293d3b32b0017a0"} Sep 16 01:29:22.758116 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.758050 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" podStartSLOduration=2.355043024 podStartE2EDuration="2.758030758s" totalImagesPullingTime="402.987734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:29:22.755636044 +0000 UTC m=+813.341438265" watchObservedRunningTime="2026-09-16 01:29:22.758030758 +0000 UTC m=+813.343832968" Sep 16 01:29:22.968037 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.967963 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2"] Sep 16 01:29:22.973078 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.973052 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:22.975476 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.975448 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-033feb-serving-cert\"" Sep 16 01:29:22.982708 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:22.982344 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2"] Sep 16 01:29:23.103777 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.103738 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-metrics-certs\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.103777 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.103778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-maas-api-tls\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.104338 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.103831 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf010e9-b807-42f3-a601-cf7b67b477c1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.104338 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.103893 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572j6\" (UniqueName: \"kubernetes.io/projected/1bf010e9-b807-42f3-a601-cf7b67b477c1-kube-api-access-572j6\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.204900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.204856 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-metrics-certs\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.205120 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.204915 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-maas-api-tls\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.205120 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.204948 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf010e9-b807-42f3-a601-cf7b67b477c1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.205120 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.204989 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572j6\" (UniqueName: \"kubernetes.io/projected/1bf010e9-b807-42f3-a601-cf7b67b477c1-kube-api-access-572j6\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.205120 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:29:23.205074 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-serving-cert: secret "maas-api-e2e-inf-a-033feb-serving-cert" not found Sep 16 01:29:23.205387 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:29:23.205147 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-maas-api-tls podName:1bf010e9-b807-42f3-a601-cf7b67b477c1 nodeName:}" failed. No retries permitted until 2026-09-16 01:29:23.70513006 +0000 UTC m=+814.290932249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-maas-api-tls") pod "maas-api-e2e-inf-a-033feb-665566884-nz7m2" (UID: "1bf010e9-b807-42f3-a601-cf7b67b477c1") : secret "maas-api-e2e-inf-a-033feb-serving-cert" not found Sep 16 01:29:23.205724 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.205705 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf010e9-b807-42f3-a601-cf7b67b477c1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.207519 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.207493 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-metrics-certs\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.213704 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.213681 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-572j6\" (UniqueName: \"kubernetes.io/projected/1bf010e9-b807-42f3-a601-cf7b67b477c1-kube-api-access-572j6\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.710940 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.710895 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-maas-api-tls\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.713539 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.713509 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-maas-api-tls\") pod \"maas-api-e2e-inf-a-033feb-665566884-nz7m2\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:23.739396 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.739359 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" event={"ID":"a9471daf-6d5c-4dff-92fb-6d8b215cda34","Type":"ContainerStarted","Data":"9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26"} Sep 16 01:29:23.739655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.739637 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:23.778633 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.778575 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" podStartSLOduration=1.416568273 podStartE2EDuration="1.778557634s" totalImagesPullingTime="361.989361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:29:23.776402981 +0000 UTC m=+814.362205196" watchObservedRunningTime="2026-09-16 01:29:23.778557634 +0000 UTC m=+814.364359863" Sep 16 01:29:23.887856 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:23.887818 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:24.165469 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:24.160780 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2"] Sep 16 01:29:24.745832 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:24.745792 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" event={"ID":"1bf010e9-b807-42f3-a601-cf7b67b477c1","Type":"ContainerStarted","Data":"76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6"} Sep 16 01:29:24.746061 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:24.745858 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" event={"ID":"1bf010e9-b807-42f3-a601-cf7b67b477c1","Type":"ContainerStarted","Data":"70e9f83ddb9bed19373ddc927592887c3e35c5e7a7f2887174ba2ad67c16fcdf"} Sep 16 01:29:24.746061 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:24.746014 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:24.763019 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:24.762964 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" podStartSLOduration=2.450399424 podStartE2EDuration="2.762945888s" totalImagesPullingTime="312.546464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:29:24.761062831 +0000 UTC m=+815.346865041" watchObservedRunningTime="2026-09-16 01:29:24.762945888 +0000 UTC m=+815.348748098" Sep 16 01:29:28.745047 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:28.745018 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:29:29.751488 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:29.751461 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:29:30.756449 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:30.756414 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:29:56.136875 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.136839 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z"] Sep 16 01:29:56.142142 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.142120 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.144687 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.144667 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a58460-serving-cert\"" Sep 16 01:29:56.148641 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.148533 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z"] Sep 16 01:29:56.218290 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.218251 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-metrics-certs\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.218466 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.218332 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.218466 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.218412 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-maas-api-tls\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.218466 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.218430 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8qbs\" (UniqueName: \"kubernetes.io/projected/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-kube-api-access-m8qbs\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.319868 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.319821 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-metrics-certs\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.320065 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.319889 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.320065 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.319969 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-maas-api-tls\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.320065 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.319995 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8qbs\" (UniqueName: \"kubernetes.io/projected/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-kube-api-access-m8qbs\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.320065 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:29:56.320049 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-serving-cert: secret "maas-api-e2e-inf-b-a58460-serving-cert" not found Sep 16 01:29:56.320243 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:29:56.320124 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-maas-api-tls podName:e8359cc9-55ca-48c3-a3ce-e63a54e4765b nodeName:}" failed. No retries permitted until 2026-09-16 01:29:56.82010804 +0000 UTC m=+847.405910229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-maas-api-tls") pod "maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" (UID: "e8359cc9-55ca-48c3-a3ce-e63a54e4765b") : secret "maas-api-e2e-inf-b-a58460-serving-cert" not found Sep 16 01:29:56.320767 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.320696 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.322574 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.322542 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-metrics-certs\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.328715 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.328692 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8qbs\" (UniqueName: \"kubernetes.io/projected/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-kube-api-access-m8qbs\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.825882 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.825842 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-maas-api-tls\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:56.828509 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:56.828481 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-maas-api-tls\") pod \"maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:57.056410 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:57.056376 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:57.326647 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:57.326618 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z"] Sep 16 01:29:57.328636 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:29:57.328596 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8359cc9_55ca_48c3_a3ce_e63a54e4765b.slice/crio-b3f0f897d45f3fc75729ff7be1bc4e2def068fb7f64b93324102aef4b1c1384d WatchSource:0}: Error finding container b3f0f897d45f3fc75729ff7be1bc4e2def068fb7f64b93324102aef4b1c1384d: Status 404 returned error can't find the container with id b3f0f897d45f3fc75729ff7be1bc4e2def068fb7f64b93324102aef4b1c1384d Sep 16 01:29:57.909770 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:57.909727 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" event={"ID":"e8359cc9-55ca-48c3-a3ce-e63a54e4765b","Type":"ContainerStarted","Data":"ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a"} Sep 16 01:29:57.909932 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:57.909799 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" event={"ID":"e8359cc9-55ca-48c3-a3ce-e63a54e4765b","Type":"ContainerStarted","Data":"b3f0f897d45f3fc75729ff7be1bc4e2def068fb7f64b93324102aef4b1c1384d"} Sep 16 01:29:57.909932 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:57.909848 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:29:57.927568 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:29:57.927469 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" podStartSLOduration=1.579512716 podStartE2EDuration="1.927452604s" totalImagesPullingTime="347.939888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:29:57.925374673 +0000 UTC m=+848.511176885" watchObservedRunningTime="2026-09-16 01:29:57.927452604 +0000 UTC m=+848.513254815" Sep 16 01:30:03.919520 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:03.919485 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:30:09.913553 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:09.913515 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9"] Sep 16 01:30:09.917095 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:09.917074 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:09.921413 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:09.921388 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d0c9f2\"/\"simulator-w3-d0c9f2-kserve-self-signed-certs\"" Sep 16 01:30:09.921534 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:09.921440 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d0c9f2\"/\"default-dockercfg-8n69k\"" Sep 16 01:30:09.921746 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:09.921728 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d0c9f2\"/\"kube-root-ca.crt\"" Sep 16 01:30:09.921851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:09.921769 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d0c9f2\"/\"openshift-service-ca.crt\"" Sep 16 01:30:09.941178 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:09.941149 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9"] Sep 16 01:30:10.059148 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.059117 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tls-certs\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.059363 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.059160 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-home\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.059363 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.059191 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-dshm\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.059363 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.059312 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwldl\" (UniqueName: \"kubernetes.io/projected/f120328d-630b-40ce-8d1a-8124ca1f3c1c-kube-api-access-mwldl\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.059553 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.059406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tmp-dir\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.059553 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.059454 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-model-cache\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.160239 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.160196 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tls-certs\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.160473 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.160242 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-home\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.160473 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.160324 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-dshm\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.160473 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.160395 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwldl\" (UniqueName: \"kubernetes.io/projected/f120328d-630b-40ce-8d1a-8124ca1f3c1c-kube-api-access-mwldl\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.160473 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.160458 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tmp-dir\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.160691 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.160499 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-model-cache\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.160691 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.160671 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-home\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.160860 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.160839 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-model-cache\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.160922 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.160857 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tmp-dir\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.162855 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.162838 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-dshm\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.162934 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.162902 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tls-certs\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.168120 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.168067 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwldl\" (UniqueName: \"kubernetes.io/projected/f120328d-630b-40ce-8d1a-8124ca1f3c1c-kube-api-access-mwldl\") pod \"simulator-w3-d0c9f2-kserve-5b496cffd4-985b9\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.227496 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.227460 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:10.502815 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.502793 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9"] Sep 16 01:30:10.504777 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:30:10.504743 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf120328d_630b_40ce_8d1a_8124ca1f3c1c.slice/crio-f34128566bc5be9042c30b36dd5cd5d69759647650749b2120feed66968c796f WatchSource:0}: Error finding container f34128566bc5be9042c30b36dd5cd5d69759647650749b2120feed66968c796f: Status 404 returned error can't find the container with id f34128566bc5be9042c30b36dd5cd5d69759647650749b2120feed66968c796f Sep 16 01:30:10.988332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.988296 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" event={"ID":"f120328d-630b-40ce-8d1a-8124ca1f3c1c","Type":"ContainerStarted","Data":"2be2d65b926d5a0d43b22db0a2960f4bdbce6ff946d75861ae17924d4ebecfce"} Sep 16 01:30:10.988711 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:10.988341 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" event={"ID":"f120328d-630b-40ce-8d1a-8124ca1f3c1c","Type":"ContainerStarted","Data":"f34128566bc5be9042c30b36dd5cd5d69759647650749b2120feed66968c796f"} Sep 16 01:30:11.004906 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:11.004852 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" podStartSLOduration=2.004836799 podStartE2EDuration="2.004836799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:30:11.002908636 +0000 UTC m=+861.588710846" watchObservedRunningTime="2026-09-16 01:30:11.004836799 +0000 UTC m=+861.590639011" Sep 16 01:30:20.227695 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:20.227653 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:20.228176 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:20.227714 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:20.239996 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:20.239972 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:21.056850 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:21.056815 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:30:35.310557 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.310521 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs"] Sep 16 01:30:35.315424 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.315401 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.318015 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.317990 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-033feb\"/\"kube-root-ca.crt\"" Sep 16 01:30:35.318147 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.318039 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-033feb\"/\"openshift-service-ca.crt\"" Sep 16 01:30:35.319926 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.319907 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-033feb\"/\"test-model-033feb-kserve-self-signed-certs\"" Sep 16 01:30:35.320668 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.320226 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-033feb\"/\"default-dockercfg-n6mt2\"" Sep 16 01:30:35.322510 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.322487 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs"] Sep 16 01:30:35.407460 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.407426 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx82h\" (UniqueName: \"kubernetes.io/projected/ed962971-2bf0-4b05-962a-ba1cc74e896b-kube-api-access-zx82h\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.407591 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.407465 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-dshm\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.407591 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.407483 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed962971-2bf0-4b05-962a-ba1cc74e896b-tls-certs\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.407707 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.407612 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-model-cache\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.407707 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.407647 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-home\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.407707 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.407680 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-tmp-dir\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.509168 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.509132 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx82h\" (UniqueName: \"kubernetes.io/projected/ed962971-2bf0-4b05-962a-ba1cc74e896b-kube-api-access-zx82h\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.509168 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.509169 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-dshm\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.509462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.509188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed962971-2bf0-4b05-962a-ba1cc74e896b-tls-certs\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.509462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.509223 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-model-cache\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.509462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.509240 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-home\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.509462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.509260 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-tmp-dir\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.509741 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.509718 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-model-cache\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.509783 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.509748 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-tmp-dir\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.509783 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.509767 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-home\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.512117 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.512099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-dshm\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.512326 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.512309 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed962971-2bf0-4b05-962a-ba1cc74e896b-tls-certs\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.516765 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.516740 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx82h\" (UniqueName: \"kubernetes.io/projected/ed962971-2bf0-4b05-962a-ba1cc74e896b-kube-api-access-zx82h\") pod \"test-model-033feb-kserve-6c79bb566c-2pdrs\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.647222 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.647129 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:35.925465 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:35.925438 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs"] Sep 16 01:30:35.927519 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:30:35.927498 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded962971_2bf0_4b05_962a_ba1cc74e896b.slice/crio-488364c2f980f78a9166f21b7ff610e00e1e285d91db68dbeedd36145d4a3af1 WatchSource:0}: Error finding container 488364c2f980f78a9166f21b7ff610e00e1e285d91db68dbeedd36145d4a3af1: Status 404 returned error can't find the container with id 488364c2f980f78a9166f21b7ff610e00e1e285d91db68dbeedd36145d4a3af1 Sep 16 01:30:36.118347 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:36.118312 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" event={"ID":"ed962971-2bf0-4b05-962a-ba1cc74e896b","Type":"ContainerStarted","Data":"65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341"} Sep 16 01:30:36.118522 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:36.118357 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" event={"ID":"ed962971-2bf0-4b05-962a-ba1cc74e896b","Type":"ContainerStarted","Data":"488364c2f980f78a9166f21b7ff610e00e1e285d91db68dbeedd36145d4a3af1"} Sep 16 01:30:36.134762 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:36.134716 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" podStartSLOduration=1.134701224 podStartE2EDuration="1.134701224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:30:36.133558429 +0000 UTC m=+886.719360641" watchObservedRunningTime="2026-09-16 01:30:36.134701224 +0000 UTC m=+886.720503434" Sep 16 01:30:45.648335 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:45.648228 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:45.648335 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:45.648301 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:45.660961 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:45.660931 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:30:46.180716 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:30:46.180690 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:32:16.800978 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.800944 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd"] Sep 16 01:32:16.804763 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.804739 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:16.807366 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.807265 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a58460\"/\"kube-root-ca.crt\"" Sep 16 01:32:16.808515 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.808486 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a58460\"/\"default-dockercfg-thhvd\"" Sep 16 01:32:16.808643 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.808509 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a58460\"/\"test-model-a58460-kserve-self-signed-certs\"" Sep 16 01:32:16.808643 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.808536 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a58460\"/\"openshift-service-ca.crt\"" Sep 16 01:32:16.814433 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.814407 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd"] Sep 16 01:32:16.905988 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.905953 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28gzf\" (UniqueName: \"kubernetes.io/projected/a36e0add-9c25-450e-add8-403064e60a6f-kube-api-access-28gzf\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:16.905988 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.905992 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-home\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:16.906208 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.906088 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-dshm\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:16.906208 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.906131 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-model-cache\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:16.906208 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.906176 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-tmp-dir\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:16.906346 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:16.906219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36e0add-9c25-450e-add8-403064e60a6f-tls-certs\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.007567 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.007532 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-dshm\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.007753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.007578 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-model-cache\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.007753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.007598 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-tmp-dir\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.007753 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.007627 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36e0add-9c25-450e-add8-403064e60a6f-tls-certs\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.007948 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.007785 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28gzf\" (UniqueName: \"kubernetes.io/projected/a36e0add-9c25-450e-add8-403064e60a6f-kube-api-access-28gzf\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.007948 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.007827 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-home\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.008049 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.007964 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-model-cache\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.008049 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.008011 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-tmp-dir\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.008155 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.008138 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-home\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.010066 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.010033 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-dshm\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.010319 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.010293 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36e0add-9c25-450e-add8-403064e60a6f-tls-certs\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.016188 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.016165 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28gzf\" (UniqueName: \"kubernetes.io/projected/a36e0add-9c25-450e-add8-403064e60a6f-kube-api-access-28gzf\") pod \"test-model-a58460-kserve-8cb5d6485-h88hd\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.115256 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.115174 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:17.390996 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.390962 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd"] Sep 16 01:32:17.392465 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:32:17.392428 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda36e0add_9c25_450e_add8_403064e60a6f.slice/crio-8b6f834b854ecef5f50219ed1326d399067f7a5a8b9e3260bf1a81b09212ba7b WatchSource:0}: Error finding container 8b6f834b854ecef5f50219ed1326d399067f7a5a8b9e3260bf1a81b09212ba7b: Status 404 returned error can't find the container with id 8b6f834b854ecef5f50219ed1326d399067f7a5a8b9e3260bf1a81b09212ba7b Sep 16 01:32:17.618701 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.618666 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" event={"ID":"a36e0add-9c25-450e-add8-403064e60a6f","Type":"ContainerStarted","Data":"7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b"} Sep 16 01:32:17.618701 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.618707 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" event={"ID":"a36e0add-9c25-450e-add8-403064e60a6f","Type":"ContainerStarted","Data":"8b6f834b854ecef5f50219ed1326d399067f7a5a8b9e3260bf1a81b09212ba7b"} Sep 16 01:32:17.635166 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:17.635112 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" podStartSLOduration=1.635096246 podStartE2EDuration="1.635096246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:32:17.633190766 +0000 UTC m=+988.218992989" watchObservedRunningTime="2026-09-16 01:32:17.635096246 +0000 UTC m=+988.220898456" Sep 16 01:32:27.116235 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:27.116192 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:27.116664 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:27.116439 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:27.129261 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:27.129236 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:32:27.686505 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:32:27.686476 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:33:29.237565 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.237521 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk"] Sep 16 01:33:29.241134 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.241116 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.243483 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.243459 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ba321c07-serving-cert\"" Sep 16 01:33:29.252449 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.252412 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk"] Sep 16 01:33:29.298456 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.298423 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6576ee7e-e520-4145-bf6c-c74a635da177-openshift-service-ca\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.298456 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.298460 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.298758 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.298519 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptr4d\" (UniqueName: \"kubernetes.io/projected/6576ee7e-e520-4145-bf6c-c74a635da177-kube-api-access-ptr4d\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.298758 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.298549 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-metrics-certs\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.399512 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.399474 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6576ee7e-e520-4145-bf6c-c74a635da177-openshift-service-ca\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.399512 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.399518 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.399781 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.399559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptr4d\" (UniqueName: \"kubernetes.io/projected/6576ee7e-e520-4145-bf6c-c74a635da177-kube-api-access-ptr4d\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.399781 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.399591 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-metrics-certs\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.399781 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:33:29.399717 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-serving-cert: secret "maas-api-e2e-ait-ba321c07-serving-cert" not found Sep 16 01:33:29.399927 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:33:29.399808 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls podName:6576ee7e-e520-4145-bf6c-c74a635da177 nodeName:}" failed. No retries permitted until 2026-09-16 01:33:29.899785724 +0000 UTC m=+1060.485587918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls") pod "maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" (UID: "6576ee7e-e520-4145-bf6c-c74a635da177") : secret "maas-api-e2e-ait-ba321c07-serving-cert" not found Sep 16 01:33:29.400164 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.400139 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6576ee7e-e520-4145-bf6c-c74a635da177-openshift-service-ca\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.402244 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.402221 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-metrics-certs\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.410290 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.410257 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptr4d\" (UniqueName: \"kubernetes.io/projected/6576ee7e-e520-4145-bf6c-c74a635da177-kube-api-access-ptr4d\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.903997 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:29.903962 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:29.904213 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:33:29.904106 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-serving-cert: secret "maas-api-e2e-ait-ba321c07-serving-cert" not found Sep 16 01:33:29.904213 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:33:29.904170 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls podName:6576ee7e-e520-4145-bf6c-c74a635da177 nodeName:}" failed. No retries permitted until 2026-09-16 01:33:30.904155944 +0000 UTC m=+1061.489958133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls") pod "maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" (UID: "6576ee7e-e520-4145-bf6c-c74a635da177") : secret "maas-api-e2e-ait-ba321c07-serving-cert" not found Sep 16 01:33:30.916535 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:30.916497 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:30.919206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:30.919172 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls\") pod \"maas-api-e2e-ait-ba321c07-f7d59794f-csnfk\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:31.054693 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:31.054660 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:31.326738 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:31.326713 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk"] Sep 16 01:33:31.328788 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:33:31.328756 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6576ee7e_e520_4145_bf6c_c74a635da177.slice/crio-fdb2bb0c96acbcbc5cc1b2c584e2a0e0342beaa55d84cc9d45339326ca68c2a1 WatchSource:0}: Error finding container fdb2bb0c96acbcbc5cc1b2c584e2a0e0342beaa55d84cc9d45339326ca68c2a1: Status 404 returned error can't find the container with id fdb2bb0c96acbcbc5cc1b2c584e2a0e0342beaa55d84cc9d45339326ca68c2a1 Sep 16 01:33:31.330044 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:31.330026 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 01:33:32.004960 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:32.004928 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" event={"ID":"6576ee7e-e520-4145-bf6c-c74a635da177","Type":"ContainerStarted","Data":"fdb2bb0c96acbcbc5cc1b2c584e2a0e0342beaa55d84cc9d45339326ca68c2a1"} Sep 16 01:33:33.011019 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:33.010971 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" event={"ID":"6576ee7e-e520-4145-bf6c-c74a635da177","Type":"ContainerStarted","Data":"e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4"} Sep 16 01:33:33.011519 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:33.011137 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:33.027998 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:33.027949 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" podStartSLOduration=3.414380772 podStartE2EDuration="4.027935054s" totalImagesPullingTime="613.554282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:33:33.026846886 +0000 UTC m=+1063.612649100" watchObservedRunningTime="2026-09-16 01:33:33.027935054 +0000 UTC m=+1063.613737265" Sep 16 01:33:39.022534 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:39.022504 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:33:49.848068 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:49.847871 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs"] Sep 16 01:33:53.653259 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:33:53.653224 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd"] Sep 16 01:34:04.459133 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.459092 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2"] Sep 16 01:34:04.459749 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.459359 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" podUID="1bf010e9-b807-42f3-a601-cf7b67b477c1" containerName="maas-api" containerID="cri-o://76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6" gracePeriod=30 Sep 16 01:34:04.709303 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.709225 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:34:04.753894 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.753860 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-maas-api-tls\") pod \"1bf010e9-b807-42f3-a601-cf7b67b477c1\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " Sep 16 01:34:04.754079 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.753956 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf010e9-b807-42f3-a601-cf7b67b477c1-openshift-service-ca\") pod \"1bf010e9-b807-42f3-a601-cf7b67b477c1\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " Sep 16 01:34:04.754079 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.754003 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-572j6\" (UniqueName: \"kubernetes.io/projected/1bf010e9-b807-42f3-a601-cf7b67b477c1-kube-api-access-572j6\") pod \"1bf010e9-b807-42f3-a601-cf7b67b477c1\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " Sep 16 01:34:04.754079 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.754049 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-metrics-certs\") pod \"1bf010e9-b807-42f3-a601-cf7b67b477c1\" (UID: \"1bf010e9-b807-42f3-a601-cf7b67b477c1\") " Sep 16 01:34:04.754485 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.754436 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf010e9-b807-42f3-a601-cf7b67b477c1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1bf010e9-b807-42f3-a601-cf7b67b477c1" (UID: "1bf010e9-b807-42f3-a601-cf7b67b477c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:34:04.756513 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.756483 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1bf010e9-b807-42f3-a601-cf7b67b477c1" (UID: "1bf010e9-b807-42f3-a601-cf7b67b477c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:34:04.756703 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.756680 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf010e9-b807-42f3-a601-cf7b67b477c1-kube-api-access-572j6" (OuterVolumeSpecName: "kube-api-access-572j6") pod "1bf010e9-b807-42f3-a601-cf7b67b477c1" (UID: "1bf010e9-b807-42f3-a601-cf7b67b477c1"). InnerVolumeSpecName "kube-api-access-572j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:04.756969 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.756942 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1bf010e9-b807-42f3-a601-cf7b67b477c1" (UID: "1bf010e9-b807-42f3-a601-cf7b67b477c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:34:04.855341 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.855306 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:04.855341 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.855340 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bf010e9-b807-42f3-a601-cf7b67b477c1-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:04.855341 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.855350 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-572j6\" (UniqueName: \"kubernetes.io/projected/1bf010e9-b807-42f3-a601-cf7b67b477c1-kube-api-access-572j6\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:04.855568 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.855358 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bf010e9-b807-42f3-a601-cf7b67b477c1-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:04.872104 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:04.872059 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" podUID="ed962971-2bf0-4b05-962a-ba1cc74e896b" containerName="main" containerID="cri-o://65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341" gracePeriod=45 Sep 16 01:34:05.109416 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.109390 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:34:05.158196 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158162 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-tmp-dir\") pod \"ed962971-2bf0-4b05-962a-ba1cc74e896b\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " Sep 16 01:34:05.158196 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158200 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-home\") pod \"ed962971-2bf0-4b05-962a-ba1cc74e896b\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " Sep 16 01:34:05.158468 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158251 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx82h\" (UniqueName: \"kubernetes.io/projected/ed962971-2bf0-4b05-962a-ba1cc74e896b-kube-api-access-zx82h\") pod \"ed962971-2bf0-4b05-962a-ba1cc74e896b\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " Sep 16 01:34:05.158468 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158271 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-dshm\") pod \"ed962971-2bf0-4b05-962a-ba1cc74e896b\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " Sep 16 01:34:05.158468 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158329 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-model-cache\") pod \"ed962971-2bf0-4b05-962a-ba1cc74e896b\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " Sep 16 01:34:05.158468 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158401 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed962971-2bf0-4b05-962a-ba1cc74e896b-tls-certs\") pod \"ed962971-2bf0-4b05-962a-ba1cc74e896b\" (UID: \"ed962971-2bf0-4b05-962a-ba1cc74e896b\") " Sep 16 01:34:05.158679 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158502 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed962971-2bf0-4b05-962a-ba1cc74e896b" (UID: "ed962971-2bf0-4b05-962a-ba1cc74e896b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:05.158679 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158551 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-home" (OuterVolumeSpecName: "home") pod "ed962971-2bf0-4b05-962a-ba1cc74e896b" (UID: "ed962971-2bf0-4b05-962a-ba1cc74e896b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:05.158679 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158628 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed962971-2bf0-4b05-962a-ba1cc74e896b" (UID: "ed962971-2bf0-4b05-962a-ba1cc74e896b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:05.158787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158739 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-home\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:05.158787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158751 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-model-cache\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:05.158787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.158761 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-tmp-dir\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:05.160700 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.160679 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed962971-2bf0-4b05-962a-ba1cc74e896b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed962971-2bf0-4b05-962a-ba1cc74e896b" (UID: "ed962971-2bf0-4b05-962a-ba1cc74e896b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:34:05.160918 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.160899 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-dshm" (OuterVolumeSpecName: "dshm") pod "ed962971-2bf0-4b05-962a-ba1cc74e896b" (UID: "ed962971-2bf0-4b05-962a-ba1cc74e896b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:05.161009 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.160922 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed962971-2bf0-4b05-962a-ba1cc74e896b-kube-api-access-zx82h" (OuterVolumeSpecName: "kube-api-access-zx82h") pod "ed962971-2bf0-4b05-962a-ba1cc74e896b" (UID: "ed962971-2bf0-4b05-962a-ba1cc74e896b"). InnerVolumeSpecName "kube-api-access-zx82h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:05.174119 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.174088 2567 generic.go:356] "Generic (PLEG): container finished" podID="ed962971-2bf0-4b05-962a-ba1cc74e896b" containerID="65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341" exitCode=0 Sep 16 01:34:05.174249 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.174146 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" Sep 16 01:34:05.174249 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.174186 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" event={"ID":"ed962971-2bf0-4b05-962a-ba1cc74e896b","Type":"ContainerDied","Data":"65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341"} Sep 16 01:34:05.174249 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.174227 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs" event={"ID":"ed962971-2bf0-4b05-962a-ba1cc74e896b","Type":"ContainerDied","Data":"488364c2f980f78a9166f21b7ff610e00e1e285d91db68dbeedd36145d4a3af1"} Sep 16 01:34:05.174249 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.174243 2567 scope.go:117] "RemoveContainer" containerID="65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341" Sep 16 01:34:05.175855 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.175831 2567 generic.go:356] "Generic (PLEG): container finished" podID="1bf010e9-b807-42f3-a601-cf7b67b477c1" containerID="76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6" exitCode=0 Sep 16 01:34:05.175997 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.175865 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" Sep 16 01:34:05.175997 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.175938 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" event={"ID":"1bf010e9-b807-42f3-a601-cf7b67b477c1","Type":"ContainerDied","Data":"76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6"} Sep 16 01:34:05.176163 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.176145 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2" event={"ID":"1bf010e9-b807-42f3-a601-cf7b67b477c1","Type":"ContainerDied","Data":"70e9f83ddb9bed19373ddc927592887c3e35c5e7a7f2887174ba2ad67c16fcdf"} Sep 16 01:34:05.184508 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.184491 2567 scope.go:117] "RemoveContainer" containerID="65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341" Sep 16 01:34:05.184814 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:34:05.184781 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341\": container with ID starting with 65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341 not found: ID does not exist" containerID="65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341" Sep 16 01:34:05.184920 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.184820 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341"} err="failed to get container status \"65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341\": rpc error: code = NotFound desc = could not find container \"65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341\": container with ID starting with 65200c8630c828626cfb324f25a190506c1df929a0e7978b3d255d5e90a50341 not found: ID does not exist" Sep 16 01:34:05.184920 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.184845 2567 scope.go:117] "RemoveContainer" containerID="76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6" Sep 16 01:34:05.193056 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.193041 2567 scope.go:117] "RemoveContainer" containerID="76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6" Sep 16 01:34:05.193317 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:34:05.193302 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6\": container with ID starting with 76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6 not found: ID does not exist" containerID="76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6" Sep 16 01:34:05.193369 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.193322 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6"} err="failed to get container status \"76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6\": rpc error: code = NotFound desc = could not find container \"76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6\": container with ID starting with 76aaca35652fe5bc132baff61e1c13923ec1bdb8ad9ab0391680d3538ddc5ef6 not found: ID does not exist" Sep 16 01:34:05.199837 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.199813 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs"] Sep 16 01:34:05.202842 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.202822 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-033feb/test-model-033feb-kserve-6c79bb566c-2pdrs"] Sep 16 01:34:05.212447 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.212422 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2"] Sep 16 01:34:05.216648 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.216627 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-033feb-665566884-nz7m2"] Sep 16 01:34:05.259544 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.259461 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed962971-2bf0-4b05-962a-ba1cc74e896b-tls-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:05.259544 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.259489 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx82h\" (UniqueName: \"kubernetes.io/projected/ed962971-2bf0-4b05-962a-ba1cc74e896b-kube-api-access-zx82h\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:05.259544 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.259501 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed962971-2bf0-4b05-962a-ba1cc74e896b-dshm\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:05.939918 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.939883 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf010e9-b807-42f3-a601-cf7b67b477c1" path="/var/lib/kubelet/pods/1bf010e9-b807-42f3-a601-cf7b67b477c1/volumes" Sep 16 01:34:05.940397 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:05.940381 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed962971-2bf0-4b05-962a-ba1cc74e896b" path="/var/lib/kubelet/pods/ed962971-2bf0-4b05-962a-ba1cc74e896b/volumes" Sep 16 01:34:07.917722 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:07.917688 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk"] Sep 16 01:34:07.918179 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:07.917949 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" podUID="6576ee7e-e520-4145-bf6c-c74a635da177" containerName="maas-api" containerID="cri-o://e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4" gracePeriod=30 Sep 16 01:34:08.169586 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.169526 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:34:08.197747 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.197710 2567 generic.go:356] "Generic (PLEG): container finished" podID="6576ee7e-e520-4145-bf6c-c74a635da177" containerID="e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4" exitCode=0 Sep 16 01:34:08.197907 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.197786 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" Sep 16 01:34:08.197907 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.197798 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" event={"ID":"6576ee7e-e520-4145-bf6c-c74a635da177","Type":"ContainerDied","Data":"e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4"} Sep 16 01:34:08.197907 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.197860 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk" event={"ID":"6576ee7e-e520-4145-bf6c-c74a635da177","Type":"ContainerDied","Data":"fdb2bb0c96acbcbc5cc1b2c584e2a0e0342beaa55d84cc9d45339326ca68c2a1"} Sep 16 01:34:08.197907 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.197889 2567 scope.go:117] "RemoveContainer" containerID="e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4" Sep 16 01:34:08.214886 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.214841 2567 scope.go:117] "RemoveContainer" containerID="e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4" Sep 16 01:34:08.215241 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:34:08.215218 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4\": container with ID starting with e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4 not found: ID does not exist" containerID="e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4" Sep 16 01:34:08.215366 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.215245 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4"} err="failed to get container status \"e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4\": rpc error: code = NotFound desc = could not find container \"e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4\": container with ID starting with e2c0b5fc47fc046e9487e6b1bdd7c3a4bcf285efeee82528508c2b961252afa4 not found: ID does not exist" Sep 16 01:34:08.292064 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.292027 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-metrics-certs\") pod \"6576ee7e-e520-4145-bf6c-c74a635da177\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " Sep 16 01:34:08.292213 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.292198 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptr4d\" (UniqueName: \"kubernetes.io/projected/6576ee7e-e520-4145-bf6c-c74a635da177-kube-api-access-ptr4d\") pod \"6576ee7e-e520-4145-bf6c-c74a635da177\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " Sep 16 01:34:08.292295 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.292231 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6576ee7e-e520-4145-bf6c-c74a635da177-openshift-service-ca\") pod \"6576ee7e-e520-4145-bf6c-c74a635da177\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " Sep 16 01:34:08.292295 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.292267 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls\") pod \"6576ee7e-e520-4145-bf6c-c74a635da177\" (UID: \"6576ee7e-e520-4145-bf6c-c74a635da177\") " Sep 16 01:34:08.292820 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.292731 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6576ee7e-e520-4145-bf6c-c74a635da177-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6576ee7e-e520-4145-bf6c-c74a635da177" (UID: "6576ee7e-e520-4145-bf6c-c74a635da177"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:34:08.294442 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.294411 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6576ee7e-e520-4145-bf6c-c74a635da177" (UID: "6576ee7e-e520-4145-bf6c-c74a635da177"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:34:08.294637 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.294612 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6576ee7e-e520-4145-bf6c-c74a635da177-kube-api-access-ptr4d" (OuterVolumeSpecName: "kube-api-access-ptr4d") pod "6576ee7e-e520-4145-bf6c-c74a635da177" (UID: "6576ee7e-e520-4145-bf6c-c74a635da177"). InnerVolumeSpecName "kube-api-access-ptr4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:08.294686 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.294652 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6576ee7e-e520-4145-bf6c-c74a635da177" (UID: "6576ee7e-e520-4145-bf6c-c74a635da177"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:34:08.393917 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.393885 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:08.393917 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.393911 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptr4d\" (UniqueName: \"kubernetes.io/projected/6576ee7e-e520-4145-bf6c-c74a635da177-kube-api-access-ptr4d\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:08.393917 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.393921 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6576ee7e-e520-4145-bf6c-c74a635da177-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:08.394139 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.393930 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6576ee7e-e520-4145-bf6c-c74a635da177-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:08.518911 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.518879 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk"] Sep 16 01:34:08.522388 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.522363 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba321c07-f7d59794f-csnfk"] Sep 16 01:34:08.674885 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.674842 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" podUID="a36e0add-9c25-450e-add8-403064e60a6f" containerName="main" containerID="cri-o://7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b" gracePeriod=45 Sep 16 01:34:08.909847 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:08.909824 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:34:09.000610 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.000574 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-model-cache\") pod \"a36e0add-9c25-450e-add8-403064e60a6f\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " Sep 16 01:34:09.000994 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.000639 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36e0add-9c25-450e-add8-403064e60a6f-tls-certs\") pod \"a36e0add-9c25-450e-add8-403064e60a6f\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " Sep 16 01:34:09.000994 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.000769 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-home\") pod \"a36e0add-9c25-450e-add8-403064e60a6f\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " Sep 16 01:34:09.000994 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.000803 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-tmp-dir\") pod \"a36e0add-9c25-450e-add8-403064e60a6f\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " Sep 16 01:34:09.000994 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.000830 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28gzf\" (UniqueName: \"kubernetes.io/projected/a36e0add-9c25-450e-add8-403064e60a6f-kube-api-access-28gzf\") pod \"a36e0add-9c25-450e-add8-403064e60a6f\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " Sep 16 01:34:09.000994 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.000855 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-dshm\") pod \"a36e0add-9c25-450e-add8-403064e60a6f\" (UID: \"a36e0add-9c25-450e-add8-403064e60a6f\") " Sep 16 01:34:09.000994 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.000898 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a36e0add-9c25-450e-add8-403064e60a6f" (UID: "a36e0add-9c25-450e-add8-403064e60a6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:09.001404 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.001027 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-home" (OuterVolumeSpecName: "home") pod "a36e0add-9c25-450e-add8-403064e60a6f" (UID: "a36e0add-9c25-450e-add8-403064e60a6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:09.001404 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.001051 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a36e0add-9c25-450e-add8-403064e60a6f" (UID: "a36e0add-9c25-450e-add8-403064e60a6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:09.001404 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.001239 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-home\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:09.001404 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.001256 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-tmp-dir\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:09.001404 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.001268 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-model-cache\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:09.003183 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.003155 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-dshm" (OuterVolumeSpecName: "dshm") pod "a36e0add-9c25-450e-add8-403064e60a6f" (UID: "a36e0add-9c25-450e-add8-403064e60a6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:34:09.003314 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.003199 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a36e0add-9c25-450e-add8-403064e60a6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a36e0add-9c25-450e-add8-403064e60a6f" (UID: "a36e0add-9c25-450e-add8-403064e60a6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:34:09.003314 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.003247 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36e0add-9c25-450e-add8-403064e60a6f-kube-api-access-28gzf" (OuterVolumeSpecName: "kube-api-access-28gzf") pod "a36e0add-9c25-450e-add8-403064e60a6f" (UID: "a36e0add-9c25-450e-add8-403064e60a6f"). InnerVolumeSpecName "kube-api-access-28gzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:09.102305 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.102182 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28gzf\" (UniqueName: \"kubernetes.io/projected/a36e0add-9c25-450e-add8-403064e60a6f-kube-api-access-28gzf\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:09.102305 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.102212 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a36e0add-9c25-450e-add8-403064e60a6f-dshm\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:09.102305 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.102222 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a36e0add-9c25-450e-add8-403064e60a6f-tls-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:09.204900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.204870 2567 generic.go:356] "Generic (PLEG): container finished" podID="a36e0add-9c25-450e-add8-403064e60a6f" containerID="7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b" exitCode=0 Sep 16 01:34:09.205070 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.204924 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" Sep 16 01:34:09.205070 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.204941 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" event={"ID":"a36e0add-9c25-450e-add8-403064e60a6f","Type":"ContainerDied","Data":"7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b"} Sep 16 01:34:09.205070 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.204980 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd" event={"ID":"a36e0add-9c25-450e-add8-403064e60a6f","Type":"ContainerDied","Data":"8b6f834b854ecef5f50219ed1326d399067f7a5a8b9e3260bf1a81b09212ba7b"} Sep 16 01:34:09.205070 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.204999 2567 scope.go:117] "RemoveContainer" containerID="7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b" Sep 16 01:34:09.215777 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.215757 2567 scope.go:117] "RemoveContainer" containerID="7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b" Sep 16 01:34:09.216491 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:34:09.216407 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b\": container with ID starting with 7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b not found: ID does not exist" containerID="7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b" Sep 16 01:34:09.216491 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.216444 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b"} err="failed to get container status \"7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b\": rpc error: code = NotFound desc = could not find container \"7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b\": container with ID starting with 7ea5b25c8b93d6d754ae621ccf1d700a72a8004ff7b95f5ee83438da0f909f6b not found: ID does not exist" Sep 16 01:34:09.236340 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.236265 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd"] Sep 16 01:34:09.240266 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.240228 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a58460/test-model-a58460-kserve-8cb5d6485-h88hd"] Sep 16 01:34:09.939105 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.939071 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6576ee7e-e520-4145-bf6c-c74a635da177" path="/var/lib/kubelet/pods/6576ee7e-e520-4145-bf6c-c74a635da177/volumes" Sep 16 01:34:09.939514 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:09.939500 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a36e0add-9c25-450e-add8-403064e60a6f" path="/var/lib/kubelet/pods/a36e0add-9c25-450e-add8-403064e60a6f/volumes" Sep 16 01:34:13.459791 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.459460 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9"] Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.459995 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed962971-2bf0-4b05-962a-ba1cc74e896b" containerName="main" Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.460010 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed962971-2bf0-4b05-962a-ba1cc74e896b" containerName="main" Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.460025 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6576ee7e-e520-4145-bf6c-c74a635da177" containerName="maas-api" Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.460030 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6576ee7e-e520-4145-bf6c-c74a635da177" containerName="maas-api" Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.460041 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf010e9-b807-42f3-a601-cf7b67b477c1" containerName="maas-api" Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.460047 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf010e9-b807-42f3-a601-cf7b67b477c1" containerName="maas-api" Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.460062 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a36e0add-9c25-450e-add8-403064e60a6f" containerName="main" Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.460068 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a36e0add-9c25-450e-add8-403064e60a6f" containerName="main" Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.460124 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf010e9-b807-42f3-a601-cf7b67b477c1" containerName="maas-api" Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.460131 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed962971-2bf0-4b05-962a-ba1cc74e896b" containerName="main" Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.460139 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a36e0add-9c25-450e-add8-403064e60a6f" containerName="main" Sep 16 01:34:13.460257 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.460145 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6576ee7e-e520-4145-bf6c-c74a635da177" containerName="maas-api" Sep 16 01:34:13.464682 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.464661 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:13.467232 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.467210 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-3f6bf6aa-serving-cert\"" Sep 16 01:34:13.476979 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.476954 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9"] Sep 16 01:34:13.545026 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.544990 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpn9w\" (UniqueName: \"kubernetes.io/projected/41c91822-14e8-48e3-839e-43714009a648-kube-api-access-rpn9w\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:13.545198 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.545031 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-metrics-certs\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:13.545198 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.545113 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-maas-api-tls\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:13.545198 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.545176 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c91822-14e8-48e3-839e-43714009a648-openshift-service-ca\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:13.645920 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.645884 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpn9w\" (UniqueName: \"kubernetes.io/projected/41c91822-14e8-48e3-839e-43714009a648-kube-api-access-rpn9w\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:13.645920 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.645927 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-metrics-certs\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:13.646147 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.645971 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-maas-api-tls\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:13.646147 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.645995 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c91822-14e8-48e3-839e-43714009a648-openshift-service-ca\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:13.646147 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:34:13.646124 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-serving-cert: secret "maas-api-e2e-migrate-3f6bf6aa-serving-cert" not found Sep 16 01:34:13.646267 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:34:13.646195 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-maas-api-tls podName:41c91822-14e8-48e3-839e-43714009a648 nodeName:}" failed. No retries permitted until 2026-09-16 01:34:14.146176937 +0000 UTC m=+1104.731979126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-maas-api-tls") pod "maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" (UID: "41c91822-14e8-48e3-839e-43714009a648") : secret "maas-api-e2e-migrate-3f6bf6aa-serving-cert" not found Sep 16 01:34:13.646688 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.646669 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c91822-14e8-48e3-839e-43714009a648-openshift-service-ca\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:13.648797 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.648774 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-metrics-certs\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:13.654430 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:13.654408 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpn9w\" (UniqueName: \"kubernetes.io/projected/41c91822-14e8-48e3-839e-43714009a648-kube-api-access-rpn9w\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:14.150814 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:14.150769 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-maas-api-tls\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:14.153499 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:14.153477 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-maas-api-tls\") pod \"maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:14.377835 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:14.377777 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:14.652884 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:14.652859 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9"] Sep 16 01:34:14.655537 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:34:14.655502 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c91822_14e8_48e3_839e_43714009a648.slice/crio-2f29f0d8596cf76230fa76b59c41077460fa7815769c47c5399027d793b6f2cc WatchSource:0}: Error finding container 2f29f0d8596cf76230fa76b59c41077460fa7815769c47c5399027d793b6f2cc: Status 404 returned error can't find the container with id 2f29f0d8596cf76230fa76b59c41077460fa7815769c47c5399027d793b6f2cc Sep 16 01:34:15.239822 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:15.239776 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" event={"ID":"41c91822-14e8-48e3-839e-43714009a648","Type":"ContainerStarted","Data":"42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836"} Sep 16 01:34:15.240003 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:15.239843 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" event={"ID":"41c91822-14e8-48e3-839e-43714009a648","Type":"ContainerStarted","Data":"2f29f0d8596cf76230fa76b59c41077460fa7815769c47c5399027d793b6f2cc"} Sep 16 01:34:15.240003 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:15.239883 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:15.257859 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:15.257798 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" podStartSLOduration=1.975077009 podStartE2EDuration="2.257781278s" totalImagesPullingTime="282.704269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:34:15.255511126 +0000 UTC m=+1105.841313337" watchObservedRunningTime="2026-09-16 01:34:15.257781278 +0000 UTC m=+1105.843583498" Sep 16 01:34:21.250592 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:21.250551 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:22.252450 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.252416 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z"] Sep 16 01:34:22.252919 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.252667 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" podUID="e8359cc9-55ca-48c3-a3ce-e63a54e4765b" containerName="maas-api" containerID="cri-o://ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a" gracePeriod=30 Sep 16 01:34:22.520205 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.520177 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:34:22.638093 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.638059 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-metrics-certs\") pod \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " Sep 16 01:34:22.638302 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.638134 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-maas-api-tls\") pod \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " Sep 16 01:34:22.638378 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.638325 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-openshift-service-ca\") pod \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " Sep 16 01:34:22.638448 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.638375 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8qbs\" (UniqueName: \"kubernetes.io/projected/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-kube-api-access-m8qbs\") pod \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\" (UID: \"e8359cc9-55ca-48c3-a3ce-e63a54e4765b\") " Sep 16 01:34:22.638775 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.638741 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e8359cc9-55ca-48c3-a3ce-e63a54e4765b" (UID: "e8359cc9-55ca-48c3-a3ce-e63a54e4765b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:34:22.639095 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.639065 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:22.640875 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.640848 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e8359cc9-55ca-48c3-a3ce-e63a54e4765b" (UID: "e8359cc9-55ca-48c3-a3ce-e63a54e4765b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:34:22.640991 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.640915 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-kube-api-access-m8qbs" (OuterVolumeSpecName: "kube-api-access-m8qbs") pod "e8359cc9-55ca-48c3-a3ce-e63a54e4765b" (UID: "e8359cc9-55ca-48c3-a3ce-e63a54e4765b"). InnerVolumeSpecName "kube-api-access-m8qbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:22.641048 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.640980 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e8359cc9-55ca-48c3-a3ce-e63a54e4765b" (UID: "e8359cc9-55ca-48c3-a3ce-e63a54e4765b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:34:22.740654 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.740610 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8qbs\" (UniqueName: \"kubernetes.io/projected/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-kube-api-access-m8qbs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:22.740654 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.740645 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:22.740654 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:22.740654 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8359cc9-55ca-48c3-a3ce-e63a54e4765b-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:23.282993 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:23.282959 2567 generic.go:356] "Generic (PLEG): container finished" podID="e8359cc9-55ca-48c3-a3ce-e63a54e4765b" containerID="ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a" exitCode=0 Sep 16 01:34:23.283497 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:23.283002 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" Sep 16 01:34:23.283497 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:23.283044 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" event={"ID":"e8359cc9-55ca-48c3-a3ce-e63a54e4765b","Type":"ContainerDied","Data":"ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a"} Sep 16 01:34:23.283497 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:23.283103 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z" event={"ID":"e8359cc9-55ca-48c3-a3ce-e63a54e4765b","Type":"ContainerDied","Data":"b3f0f897d45f3fc75729ff7be1bc4e2def068fb7f64b93324102aef4b1c1384d"} Sep 16 01:34:23.283497 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:23.283136 2567 scope.go:117] "RemoveContainer" containerID="ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a" Sep 16 01:34:23.292969 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:23.292948 2567 scope.go:117] "RemoveContainer" containerID="ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a" Sep 16 01:34:23.293247 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:34:23.293226 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a\": container with ID starting with ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a not found: ID does not exist" containerID="ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a" Sep 16 01:34:23.293346 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:23.293256 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a"} err="failed to get container status \"ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a\": rpc error: code = NotFound desc = could not find container \"ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a\": container with ID starting with ed59b83161c6207924bb7d439b7acd34b8b284ad4634b79994417a2cfe63af6a not found: ID does not exist" Sep 16 01:34:23.305478 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:23.305441 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z"] Sep 16 01:34:23.309171 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:23.309151 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a58460-66d8b85fb4-bdt9z"] Sep 16 01:34:23.939180 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:23.939144 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8359cc9-55ca-48c3-a3ce-e63a54e4765b" path="/var/lib/kubelet/pods/e8359cc9-55ca-48c3-a3ce-e63a54e4765b/volumes" Sep 16 01:34:35.851694 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:35.851398 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws"] Sep 16 01:34:35.852356 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:35.852034 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8359cc9-55ca-48c3-a3ce-e63a54e4765b" containerName="maas-api" Sep 16 01:34:35.852356 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:35.852049 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8359cc9-55ca-48c3-a3ce-e63a54e4765b" containerName="maas-api" Sep 16 01:34:35.852356 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:35.852147 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8359cc9-55ca-48c3-a3ce-e63a54e4765b" containerName="maas-api" Sep 16 01:34:35.857236 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:35.857211 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:35.859512 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:35.859487 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-cbb499-serving-cert\"" Sep 16 01:34:35.864303 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:35.864211 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws"] Sep 16 01:34:35.959081 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:35.959037 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwnj\" (UniqueName: \"kubernetes.io/projected/fa1c0515-cf74-4231-9141-8028cc2a601f-kube-api-access-cfwnj\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:35.959258 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:35.959094 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:35.959258 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:35.959123 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:35.959258 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:35.959149 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1c0515-cf74-4231-9141-8028cc2a601f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:36.060525 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:36.060486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:36.060525 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:36.060530 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1c0515-cf74-4231-9141-8028cc2a601f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:36.060762 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:36.060631 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwnj\" (UniqueName: \"kubernetes.io/projected/fa1c0515-cf74-4231-9141-8028cc2a601f-kube-api-access-cfwnj\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:36.060762 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:36.060662 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:36.061400 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:36.061324 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1c0515-cf74-4231-9141-8028cc2a601f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:36.063163 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:36.063144 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:36.063350 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:36.063327 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:36.069115 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:36.069092 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwnj\" (UniqueName: \"kubernetes.io/projected/fa1c0515-cf74-4231-9141-8028cc2a601f-kube-api-access-cfwnj\") pod \"maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:36.173109 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:36.173024 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:36.451109 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:36.451085 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws"] Sep 16 01:34:36.453498 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:34:36.453464 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1c0515_cf74_4231_9141_8028cc2a601f.slice/crio-02636e308c0aae43d63bf27e98ad2df6a74f5dd6179fb57d1e890ded6b504fcc WatchSource:0}: Error finding container 02636e308c0aae43d63bf27e98ad2df6a74f5dd6179fb57d1e890ded6b504fcc: Status 404 returned error can't find the container with id 02636e308c0aae43d63bf27e98ad2df6a74f5dd6179fb57d1e890ded6b504fcc Sep 16 01:34:37.355928 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.355890 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" event={"ID":"fa1c0515-cf74-4231-9141-8028cc2a601f","Type":"ContainerStarted","Data":"6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32"} Sep 16 01:34:37.356395 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.355942 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" event={"ID":"fa1c0515-cf74-4231-9141-8028cc2a601f","Type":"ContainerStarted","Data":"02636e308c0aae43d63bf27e98ad2df6a74f5dd6179fb57d1e890ded6b504fcc"} Sep 16 01:34:37.356395 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.356023 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:37.373663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.373612 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" podStartSLOduration=1.9698480489999999 podStartE2EDuration="2.373595497s" totalImagesPullingTime="403.747448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:34:37.371894364 +0000 UTC m=+1127.957696579" watchObservedRunningTime="2026-09-16 01:34:37.373595497 +0000 UTC m=+1127.959397729" Sep 16 01:34:37.511331 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.511261 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9"] Sep 16 01:34:37.511514 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.511493 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" podUID="41c91822-14e8-48e3-839e-43714009a648" containerName="maas-api" containerID="cri-o://42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836" gracePeriod=30 Sep 16 01:34:37.761574 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.761548 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:37.779178 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.779124 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-maas-api-tls\") pod \"41c91822-14e8-48e3-839e-43714009a648\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " Sep 16 01:34:37.779418 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.779260 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c91822-14e8-48e3-839e-43714009a648-openshift-service-ca\") pod \"41c91822-14e8-48e3-839e-43714009a648\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " Sep 16 01:34:37.779418 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.779325 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpn9w\" (UniqueName: \"kubernetes.io/projected/41c91822-14e8-48e3-839e-43714009a648-kube-api-access-rpn9w\") pod \"41c91822-14e8-48e3-839e-43714009a648\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " Sep 16 01:34:37.779418 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.779349 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-metrics-certs\") pod \"41c91822-14e8-48e3-839e-43714009a648\" (UID: \"41c91822-14e8-48e3-839e-43714009a648\") " Sep 16 01:34:37.779691 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.779664 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c91822-14e8-48e3-839e-43714009a648-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "41c91822-14e8-48e3-839e-43714009a648" (UID: "41c91822-14e8-48e3-839e-43714009a648"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:34:37.783046 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.783012 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "41c91822-14e8-48e3-839e-43714009a648" (UID: "41c91822-14e8-48e3-839e-43714009a648"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:34:37.784253 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.784223 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "41c91822-14e8-48e3-839e-43714009a648" (UID: "41c91822-14e8-48e3-839e-43714009a648"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:34:37.784572 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.784538 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c91822-14e8-48e3-839e-43714009a648-kube-api-access-rpn9w" (OuterVolumeSpecName: "kube-api-access-rpn9w") pod "41c91822-14e8-48e3-839e-43714009a648" (UID: "41c91822-14e8-48e3-839e-43714009a648"). InnerVolumeSpecName "kube-api-access-rpn9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:34:37.880623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.880535 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41c91822-14e8-48e3-839e-43714009a648-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:37.880623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.880565 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpn9w\" (UniqueName: \"kubernetes.io/projected/41c91822-14e8-48e3-839e-43714009a648-kube-api-access-rpn9w\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:37.880623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.880575 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:37.880623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:37.880583 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41c91822-14e8-48e3-839e-43714009a648-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:34:38.362683 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:38.362645 2567 generic.go:356] "Generic (PLEG): container finished" podID="41c91822-14e8-48e3-839e-43714009a648" containerID="42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836" exitCode=0 Sep 16 01:34:38.362683 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:38.362685 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" Sep 16 01:34:38.363158 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:38.362726 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" event={"ID":"41c91822-14e8-48e3-839e-43714009a648","Type":"ContainerDied","Data":"42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836"} Sep 16 01:34:38.363158 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:38.362773 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9" event={"ID":"41c91822-14e8-48e3-839e-43714009a648","Type":"ContainerDied","Data":"2f29f0d8596cf76230fa76b59c41077460fa7815769c47c5399027d793b6f2cc"} Sep 16 01:34:38.363158 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:38.362797 2567 scope.go:117] "RemoveContainer" containerID="42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836" Sep 16 01:34:38.372378 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:38.372361 2567 scope.go:117] "RemoveContainer" containerID="42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836" Sep 16 01:34:38.372670 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:34:38.372649 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836\": container with ID starting with 42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836 not found: ID does not exist" containerID="42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836" Sep 16 01:34:38.372720 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:38.372679 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836"} err="failed to get container status \"42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836\": rpc error: code = NotFound desc = could not find container \"42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836\": container with ID starting with 42ad0209727c4f804657488a0aab0bc09efa9d2b35e4689d73080f3b486ce836 not found: ID does not exist" Sep 16 01:34:38.378704 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:38.378682 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9"] Sep 16 01:34:38.384462 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:38.384433 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-3f6bf6aa-86dd4dfd66-vzsz9"] Sep 16 01:34:39.939266 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:39.939231 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41c91822-14e8-48e3-839e-43714009a648" path="/var/lib/kubelet/pods/41c91822-14e8-48e3-839e-43714009a648/volumes" Sep 16 01:34:42.627509 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.627195 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv"] Sep 16 01:34:42.627972 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.627839 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c91822-14e8-48e3-839e-43714009a648" containerName="maas-api" Sep 16 01:34:42.627972 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.627859 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c91822-14e8-48e3-839e-43714009a648" containerName="maas-api" Sep 16 01:34:42.628097 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.627973 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c91822-14e8-48e3-839e-43714009a648" containerName="maas-api" Sep 16 01:34:42.632677 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.632653 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.634914 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.634891 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-60623346-serving-cert\"" Sep 16 01:34:42.640735 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.640710 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv"] Sep 16 01:34:42.731831 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.731793 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-metrics-certs\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.732017 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.731835 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-maas-api-tls\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.732017 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.731856 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxnhp\" (UniqueName: \"kubernetes.io/projected/708cc75a-8744-422d-aad6-b241ee29f840-kube-api-access-xxnhp\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.732017 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.731895 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708cc75a-8744-422d-aad6-b241ee29f840-openshift-service-ca\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.832610 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.832568 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708cc75a-8744-422d-aad6-b241ee29f840-openshift-service-ca\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.832816 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.832696 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-metrics-certs\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.832816 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.832718 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-maas-api-tls\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.832816 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.832733 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxnhp\" (UniqueName: \"kubernetes.io/projected/708cc75a-8744-422d-aad6-b241ee29f840-kube-api-access-xxnhp\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.833333 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.833308 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708cc75a-8744-422d-aad6-b241ee29f840-openshift-service-ca\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.835333 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.835308 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-maas-api-tls\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.835435 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.835417 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-metrics-certs\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.842968 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.842943 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxnhp\" (UniqueName: \"kubernetes.io/projected/708cc75a-8744-422d-aad6-b241ee29f840-kube-api-access-xxnhp\") pod \"maas-api-e2e-ait-60623346-7b99f7b744-p6sgv\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:42.947242 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:42.947152 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:43.233070 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:43.233043 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv"] Sep 16 01:34:43.235256 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:34:43.235230 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708cc75a_8744_422d_aad6_b241ee29f840.slice/crio-b3e44f64ec79376a151c8da332ebeb6d3b0bb7afc4cca712a0bc10c4dad22ed8 WatchSource:0}: Error finding container b3e44f64ec79376a151c8da332ebeb6d3b0bb7afc4cca712a0bc10c4dad22ed8: Status 404 returned error can't find the container with id b3e44f64ec79376a151c8da332ebeb6d3b0bb7afc4cca712a0bc10c4dad22ed8 Sep 16 01:34:43.368608 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:43.368577 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:34:43.391384 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:43.391344 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" event={"ID":"708cc75a-8744-422d-aad6-b241ee29f840","Type":"ContainerStarted","Data":"b3e44f64ec79376a151c8da332ebeb6d3b0bb7afc4cca712a0bc10c4dad22ed8"} Sep 16 01:34:44.398495 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:44.398454 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" event={"ID":"708cc75a-8744-422d-aad6-b241ee29f840","Type":"ContainerStarted","Data":"c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca"} Sep 16 01:34:44.398900 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:44.398610 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:34:44.416902 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:44.416846 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" podStartSLOduration=1.9388828980000001 podStartE2EDuration="2.416831566s" totalImagesPullingTime="477.948668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:34:44.415572635 +0000 UTC m=+1135.001374846" watchObservedRunningTime="2026-09-16 01:34:44.416831566 +0000 UTC m=+1135.002633778" Sep 16 01:34:50.409904 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:34:50.409829 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:35:00.629001 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.628963 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv"] Sep 16 01:35:00.632835 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.632816 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.635490 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.635467 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d0c9f2\"/\"distinct-2-w3-d0c9f2-kserve-self-signed-certs\"" Sep 16 01:35:00.640311 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.640086 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv"] Sep 16 01:35:00.703893 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.703858 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-home\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.704033 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.703903 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13839e18-47af-455c-9aba-6be1cefc75cd-tls-certs\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.704033 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.703972 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-dshm\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.704099 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.704067 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-tmp-dir\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.704205 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.704096 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcj62\" (UniqueName: \"kubernetes.io/projected/13839e18-47af-455c-9aba-6be1cefc75cd-kube-api-access-dcj62\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.704205 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.704159 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-model-cache\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.804789 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.804751 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-dshm\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.804980 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.804837 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-tmp-dir\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.804980 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.804930 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcj62\" (UniqueName: \"kubernetes.io/projected/13839e18-47af-455c-9aba-6be1cefc75cd-kube-api-access-dcj62\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.805101 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.804994 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-model-cache\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.805101 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.805043 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-home\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.805101 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.805084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13839e18-47af-455c-9aba-6be1cefc75cd-tls-certs\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.805313 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.805266 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-tmp-dir\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.805416 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.805393 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-model-cache\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.805483 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.805462 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-home\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.807293 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.807253 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-dshm\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.807687 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.807667 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13839e18-47af-455c-9aba-6be1cefc75cd-tls-certs\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.814380 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.814349 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcj62\" (UniqueName: \"kubernetes.io/projected/13839e18-47af-455c-9aba-6be1cefc75cd-kube-api-access-dcj62\") pod \"distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:00.943604 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:00.943515 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:01.215861 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:01.215831 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv"] Sep 16 01:35:01.217943 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:35:01.217916 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13839e18_47af_455c_9aba_6be1cefc75cd.slice/crio-ddb7fdcc6b1f8022c64c156a123dbeaa66d999ef73f78526b3cf5c8547dd63e8 WatchSource:0}: Error finding container ddb7fdcc6b1f8022c64c156a123dbeaa66d999ef73f78526b3cf5c8547dd63e8: Status 404 returned error can't find the container with id ddb7fdcc6b1f8022c64c156a123dbeaa66d999ef73f78526b3cf5c8547dd63e8 Sep 16 01:35:01.485332 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:01.485293 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" event={"ID":"13839e18-47af-455c-9aba-6be1cefc75cd","Type":"ContainerStarted","Data":"8e5a701b7075569dd9e607d586412e23f883e9ef17e20c245dd96b9788f74e01"} Sep 16 01:35:01.485482 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:01.485348 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" event={"ID":"13839e18-47af-455c-9aba-6be1cefc75cd","Type":"ContainerStarted","Data":"ddb7fdcc6b1f8022c64c156a123dbeaa66d999ef73f78526b3cf5c8547dd63e8"} Sep 16 01:35:01.501624 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:01.501566 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" podStartSLOduration=1.501549012 podStartE2EDuration="1.501549012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:35:01.500249654 +0000 UTC m=+1152.086051865" watchObservedRunningTime="2026-09-16 01:35:01.501549012 +0000 UTC m=+1152.087351261" Sep 16 01:35:05.075695 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.075643 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7"] Sep 16 01:35:05.079866 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.079841 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.082480 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.082459 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c0ec73-serving-cert\"" Sep 16 01:35:05.087303 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.087258 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7"] Sep 16 01:35:05.254592 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.254547 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.254592 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.254594 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6390385-8fb5-4c70-b967-fc444957e86f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.254829 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.254617 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.254829 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.254680 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xfk\" (UniqueName: \"kubernetes.io/projected/e6390385-8fb5-4c70-b967-fc444957e86f-kube-api-access-m7xfk\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.356238 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.356136 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.356238 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.356188 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6390385-8fb5-4c70-b967-fc444957e86f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.356238 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.356218 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.356586 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.356317 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xfk\" (UniqueName: \"kubernetes.io/projected/e6390385-8fb5-4c70-b967-fc444957e86f-kube-api-access-m7xfk\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.356944 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.356923 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6390385-8fb5-4c70-b967-fc444957e86f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.358882 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.358857 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.359012 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.358993 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.364164 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.364145 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xfk\" (UniqueName: \"kubernetes.io/projected/e6390385-8fb5-4c70-b967-fc444957e86f-kube-api-access-m7xfk\") pod \"maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.393739 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.393705 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:05.677083 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:05.677052 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7"] Sep 16 01:35:05.679699 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:35:05.679664 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6390385_8fb5_4c70_b967_fc444957e86f.slice/crio-a7f95847e13d3f3a2538cea92a0443b6e22d627ecb5f0b90786116cf1e3b3462 WatchSource:0}: Error finding container a7f95847e13d3f3a2538cea92a0443b6e22d627ecb5f0b90786116cf1e3b3462: Status 404 returned error can't find the container with id a7f95847e13d3f3a2538cea92a0443b6e22d627ecb5f0b90786116cf1e3b3462 Sep 16 01:35:06.512083 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:06.512047 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" event={"ID":"e6390385-8fb5-4c70-b967-fc444957e86f","Type":"ContainerStarted","Data":"619a150b91884dcdacf292980af83c80eea348a2e99cc461d3dff30e55830a85"} Sep 16 01:35:06.512525 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:06.512101 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" event={"ID":"e6390385-8fb5-4c70-b967-fc444957e86f","Type":"ContainerStarted","Data":"a7f95847e13d3f3a2538cea92a0443b6e22d627ecb5f0b90786116cf1e3b3462"} Sep 16 01:35:06.512525 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:06.512165 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:06.531217 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:06.531164 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" podStartSLOduration=1.235583248 podStartE2EDuration="1.531147928s" totalImagesPullingTime="295.56468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:35:06.529510679 +0000 UTC m=+1157.115312913" watchObservedRunningTime="2026-09-16 01:35:06.531147928 +0000 UTC m=+1157.116950139" Sep 16 01:35:10.944222 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:10.944140 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:10.944222 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:10.944193 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:10.956885 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:10.956856 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:11.548933 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:11.548899 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:35:12.260079 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.260038 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv"] Sep 16 01:35:12.260594 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.260243 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" podUID="708cc75a-8744-422d-aad6-b241ee29f840" containerName="maas-api" containerID="cri-o://c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca" gracePeriod=30 Sep 16 01:35:12.518690 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.518615 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:35:12.522656 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.522632 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:35:12.525286 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.525241 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708cc75a-8744-422d-aad6-b241ee29f840-openshift-service-ca\") pod \"708cc75a-8744-422d-aad6-b241ee29f840\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " Sep 16 01:35:12.525405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.525296 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-metrics-certs\") pod \"708cc75a-8744-422d-aad6-b241ee29f840\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " Sep 16 01:35:12.525405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.525335 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxnhp\" (UniqueName: \"kubernetes.io/projected/708cc75a-8744-422d-aad6-b241ee29f840-kube-api-access-xxnhp\") pod \"708cc75a-8744-422d-aad6-b241ee29f840\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " Sep 16 01:35:12.525405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.525382 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-maas-api-tls\") pod \"708cc75a-8744-422d-aad6-b241ee29f840\" (UID: \"708cc75a-8744-422d-aad6-b241ee29f840\") " Sep 16 01:35:12.525677 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.525638 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708cc75a-8744-422d-aad6-b241ee29f840-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "708cc75a-8744-422d-aad6-b241ee29f840" (UID: "708cc75a-8744-422d-aad6-b241ee29f840"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:35:12.525811 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.525782 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708cc75a-8744-422d-aad6-b241ee29f840-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:35:12.527797 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.527770 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "708cc75a-8744-422d-aad6-b241ee29f840" (UID: "708cc75a-8744-422d-aad6-b241ee29f840"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:35:12.527897 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.527813 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "708cc75a-8744-422d-aad6-b241ee29f840" (UID: "708cc75a-8744-422d-aad6-b241ee29f840"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:35:12.527897 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.527830 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708cc75a-8744-422d-aad6-b241ee29f840-kube-api-access-xxnhp" (OuterVolumeSpecName: "kube-api-access-xxnhp") pod "708cc75a-8744-422d-aad6-b241ee29f840" (UID: "708cc75a-8744-422d-aad6-b241ee29f840"). InnerVolumeSpecName "kube-api-access-xxnhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:35:12.545197 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.545167 2567 generic.go:356] "Generic (PLEG): container finished" podID="708cc75a-8744-422d-aad6-b241ee29f840" containerID="c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca" exitCode=0 Sep 16 01:35:12.545366 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.545319 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" event={"ID":"708cc75a-8744-422d-aad6-b241ee29f840","Type":"ContainerDied","Data":"c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca"} Sep 16 01:35:12.545444 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.545385 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" event={"ID":"708cc75a-8744-422d-aad6-b241ee29f840","Type":"ContainerDied","Data":"b3e44f64ec79376a151c8da332ebeb6d3b0bb7afc4cca712a0bc10c4dad22ed8"} Sep 16 01:35:12.545444 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.545414 2567 scope.go:117] "RemoveContainer" containerID="c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca" Sep 16 01:35:12.545554 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.545310 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv" Sep 16 01:35:12.556205 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.556182 2567 scope.go:117] "RemoveContainer" containerID="c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca" Sep 16 01:35:12.556572 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:35:12.556553 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca\": container with ID starting with c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca not found: ID does not exist" containerID="c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca" Sep 16 01:35:12.556661 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.556581 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca"} err="failed to get container status \"c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca\": rpc error: code = NotFound desc = could not find container \"c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca\": container with ID starting with c7c4ff1071cad995ea8f23a63994835dec8a3464a58276c31d9288aaa2434eca not found: ID does not exist" Sep 16 01:35:12.569887 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.569854 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv"] Sep 16 01:35:12.575717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.575691 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-60623346-7b99f7b744-p6sgv"] Sep 16 01:35:12.627442 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.627409 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:35:12.627442 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.627445 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxnhp\" (UniqueName: \"kubernetes.io/projected/708cc75a-8744-422d-aad6-b241ee29f840-kube-api-access-xxnhp\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:35:12.627635 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:12.627462 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708cc75a-8744-422d-aad6-b241ee29f840-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:35:13.939891 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:13.939848 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="708cc75a-8744-422d-aad6-b241ee29f840" path="/var/lib/kubelet/pods/708cc75a-8744-422d-aad6-b241ee29f840/volumes" Sep 16 01:35:19.737405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.736069 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8"] Sep 16 01:35:19.737405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.736742 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="708cc75a-8744-422d-aad6-b241ee29f840" containerName="maas-api" Sep 16 01:35:19.737405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.736765 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="708cc75a-8744-422d-aad6-b241ee29f840" containerName="maas-api" Sep 16 01:35:19.737405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.736889 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="708cc75a-8744-422d-aad6-b241ee29f840" containerName="maas-api" Sep 16 01:35:19.740357 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.740334 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.742698 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.742674 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-74a27a74-serving-cert\"" Sep 16 01:35:19.750686 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.750595 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8"] Sep 16 01:35:19.799773 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.799731 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-maas-api-tls\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.799947 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.799787 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-metrics-certs\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.799947 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.799863 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12bfceb9-f8cb-43aa-9529-7040d568b8b1-openshift-service-ca\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.799947 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.799887 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb6cf\" (UniqueName: \"kubernetes.io/projected/12bfceb9-f8cb-43aa-9529-7040d568b8b1-kube-api-access-xb6cf\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.900991 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.900952 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-maas-api-tls\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.900991 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.900995 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-metrics-certs\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.901299 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.901132 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12bfceb9-f8cb-43aa-9529-7040d568b8b1-openshift-service-ca\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.901299 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.901186 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb6cf\" (UniqueName: \"kubernetes.io/projected/12bfceb9-f8cb-43aa-9529-7040d568b8b1-kube-api-access-xb6cf\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.901830 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.901805 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12bfceb9-f8cb-43aa-9529-7040d568b8b1-openshift-service-ca\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.903621 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.903603 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-metrics-certs\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.903702 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.903647 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-maas-api-tls\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:19.908684 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:19.908658 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb6cf\" (UniqueName: \"kubernetes.io/projected/12bfceb9-f8cb-43aa-9529-7040d568b8b1-kube-api-access-xb6cf\") pod \"maas-api-e2e-derive-74a27a74-69777d7644-j5hw8\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:20.054489 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:20.054407 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:20.330049 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:20.330024 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8"] Sep 16 01:35:20.331667 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:35:20.331642 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bfceb9_f8cb_43aa_9529_7040d568b8b1.slice/crio-f1df171c85bee5f2e4fc897dd2d06636cfa1875458e3b914500615319ea52c39 WatchSource:0}: Error finding container f1df171c85bee5f2e4fc897dd2d06636cfa1875458e3b914500615319ea52c39: Status 404 returned error can't find the container with id f1df171c85bee5f2e4fc897dd2d06636cfa1875458e3b914500615319ea52c39 Sep 16 01:35:20.594744 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:20.594652 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" event={"ID":"12bfceb9-f8cb-43aa-9529-7040d568b8b1","Type":"ContainerStarted","Data":"f1df171c85bee5f2e4fc897dd2d06636cfa1875458e3b914500615319ea52c39"} Sep 16 01:35:21.602434 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:21.602393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" event={"ID":"12bfceb9-f8cb-43aa-9529-7040d568b8b1","Type":"ContainerStarted","Data":"c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb"} Sep 16 01:35:21.602799 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:21.602524 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:21.621243 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:21.621185 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" podStartSLOduration=2.307907449 podStartE2EDuration="2.621167324s" totalImagesPullingTime="313.259875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:35:21.619982284 +0000 UTC m=+1172.205784499" watchObservedRunningTime="2026-09-16 01:35:21.621167324 +0000 UTC m=+1172.206969536" Sep 16 01:35:27.613214 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:27.613182 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:49.165654 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.165616 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8"] Sep 16 01:35:49.166225 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.165883 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" podUID="12bfceb9-f8cb-43aa-9529-7040d568b8b1" containerName="maas-api" containerID="cri-o://c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb" gracePeriod=30 Sep 16 01:35:49.424706 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.424640 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:49.504997 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.504965 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb6cf\" (UniqueName: \"kubernetes.io/projected/12bfceb9-f8cb-43aa-9529-7040d568b8b1-kube-api-access-xb6cf\") pod \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " Sep 16 01:35:49.505178 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.505056 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12bfceb9-f8cb-43aa-9529-7040d568b8b1-openshift-service-ca\") pod \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " Sep 16 01:35:49.505178 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.505088 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-maas-api-tls\") pod \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " Sep 16 01:35:49.505178 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.505112 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-metrics-certs\") pod \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\" (UID: \"12bfceb9-f8cb-43aa-9529-7040d568b8b1\") " Sep 16 01:35:49.505508 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.505477 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12bfceb9-f8cb-43aa-9529-7040d568b8b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12bfceb9-f8cb-43aa-9529-7040d568b8b1" (UID: "12bfceb9-f8cb-43aa-9529-7040d568b8b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:35:49.508945 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.508921 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12bfceb9-f8cb-43aa-9529-7040d568b8b1" (UID: "12bfceb9-f8cb-43aa-9529-7040d568b8b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:35:49.509461 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.509441 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "12bfceb9-f8cb-43aa-9529-7040d568b8b1" (UID: "12bfceb9-f8cb-43aa-9529-7040d568b8b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:35:49.509550 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.509534 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bfceb9-f8cb-43aa-9529-7040d568b8b1-kube-api-access-xb6cf" (OuterVolumeSpecName: "kube-api-access-xb6cf") pod "12bfceb9-f8cb-43aa-9529-7040d568b8b1" (UID: "12bfceb9-f8cb-43aa-9529-7040d568b8b1"). InnerVolumeSpecName "kube-api-access-xb6cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:35:49.606362 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.606325 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb6cf\" (UniqueName: \"kubernetes.io/projected/12bfceb9-f8cb-43aa-9529-7040d568b8b1-kube-api-access-xb6cf\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:35:49.606362 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.606355 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12bfceb9-f8cb-43aa-9529-7040d568b8b1-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:35:49.606362 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.606365 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:35:49.606592 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.606375 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bfceb9-f8cb-43aa-9529-7040d568b8b1-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:35:49.748201 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.748166 2567 generic.go:356] "Generic (PLEG): container finished" podID="12bfceb9-f8cb-43aa-9529-7040d568b8b1" containerID="c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb" exitCode=0 Sep 16 01:35:49.748422 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.748213 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" Sep 16 01:35:49.748422 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.748254 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" event={"ID":"12bfceb9-f8cb-43aa-9529-7040d568b8b1","Type":"ContainerDied","Data":"c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb"} Sep 16 01:35:49.748422 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.748347 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8" event={"ID":"12bfceb9-f8cb-43aa-9529-7040d568b8b1","Type":"ContainerDied","Data":"f1df171c85bee5f2e4fc897dd2d06636cfa1875458e3b914500615319ea52c39"} Sep 16 01:35:49.748422 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.748388 2567 scope.go:117] "RemoveContainer" containerID="c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb" Sep 16 01:35:49.758825 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.758802 2567 scope.go:117] "RemoveContainer" containerID="c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb" Sep 16 01:35:49.759118 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:35:49.759101 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb\": container with ID starting with c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb not found: ID does not exist" containerID="c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb" Sep 16 01:35:49.759167 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.759130 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb"} err="failed to get container status \"c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb\": rpc error: code = NotFound desc = could not find container \"c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb\": container with ID starting with c4dcea7e072d965852b82474be491df7bf0d722121b6a81008968850a0be87bb not found: ID does not exist" Sep 16 01:35:49.772072 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.772047 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8"] Sep 16 01:35:49.775839 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.775818 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-74a27a74-69777d7644-j5hw8"] Sep 16 01:35:49.941008 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:35:49.940973 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12bfceb9-f8cb-43aa-9529-7040d568b8b1" path="/var/lib/kubelet/pods/12bfceb9-f8cb-43aa-9529-7040d568b8b1/volumes" Sep 16 01:36:49.016446 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.016369 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9"] Sep 16 01:36:49.018509 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.016851 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12bfceb9-f8cb-43aa-9529-7040d568b8b1" containerName="maas-api" Sep 16 01:36:49.018509 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.016872 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="12bfceb9-f8cb-43aa-9529-7040d568b8b1" containerName="maas-api" Sep 16 01:36:49.018509 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.016966 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="12bfceb9-f8cb-43aa-9529-7040d568b8b1" containerName="maas-api" Sep 16 01:36:49.019759 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.019744 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.022095 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.022074 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-cbb499\"/\"kube-root-ca.crt\"" Sep 16 01:36:49.023109 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.023092 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-cbb499\"/\"openshift-service-ca.crt\"" Sep 16 01:36:49.023344 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.023330 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-cbb499\"/\"auth-test-model-cbb499-kserve-self-signed-certs\"" Sep 16 01:36:49.023395 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.023371 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-cbb499\"/\"default-dockercfg-n6xh2\"" Sep 16 01:36:49.029440 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.029418 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9"] Sep 16 01:36:49.093213 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.093172 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-model-cache\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.093399 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.093312 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxmm\" (UniqueName: \"kubernetes.io/projected/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-kube-api-access-2wxmm\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.093471 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.093426 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-dshm\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.093529 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.093491 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tls-certs\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.093612 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.093598 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-home\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.093687 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.093671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tmp-dir\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.194448 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.194404 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxmm\" (UniqueName: \"kubernetes.io/projected/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-kube-api-access-2wxmm\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.194448 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.194442 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-dshm\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.194665 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.194481 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tls-certs\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.194665 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.194531 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-home\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.194665 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.194572 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tmp-dir\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.194665 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.194624 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-model-cache\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.195049 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.195014 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-home\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.195049 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.195033 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tmp-dir\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.195428 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.195065 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-model-cache\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.196945 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.196923 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-dshm\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.197364 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.197344 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tls-certs\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.201962 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.201932 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxmm\" (UniqueName: \"kubernetes.io/projected/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-kube-api-access-2wxmm\") pod \"auth-test-model-cbb499-kserve-59b8f965f5-lq8c9\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.329729 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.329614 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:49.601188 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:49.601162 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9"] Sep 16 01:36:49.602833 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:36:49.602803 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddf96e2d_efcd_4be9_86f2_4219f32ebb39.slice/crio-2b7e10ac9d061bef97ea7336e44b693dee67e9434869444ced550eba1a0eb2f8 WatchSource:0}: Error finding container 2b7e10ac9d061bef97ea7336e44b693dee67e9434869444ced550eba1a0eb2f8: Status 404 returned error can't find the container with id 2b7e10ac9d061bef97ea7336e44b693dee67e9434869444ced550eba1a0eb2f8 Sep 16 01:36:50.050203 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:50.050161 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" event={"ID":"ddf96e2d-efcd-4be9-86f2-4219f32ebb39","Type":"ContainerStarted","Data":"b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e"} Sep 16 01:36:50.050602 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:50.050209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" event={"ID":"ddf96e2d-efcd-4be9-86f2-4219f32ebb39","Type":"ContainerStarted","Data":"2b7e10ac9d061bef97ea7336e44b693dee67e9434869444ced550eba1a0eb2f8"} Sep 16 01:36:50.065438 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:50.065380 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" podStartSLOduration=1.065359764 podStartE2EDuration="1.065359764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:36:50.064777355 +0000 UTC m=+1260.650579566" watchObservedRunningTime="2026-09-16 01:36:50.065359764 +0000 UTC m=+1260.651161976" Sep 16 01:36:59.330297 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:59.330232 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:59.330297 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:59.330301 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:36:59.342814 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:36:59.342782 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:37:00.113850 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:00.113809 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:37:16.260130 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.260079 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt"] Sep 16 01:37:16.260620 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.260369 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" podUID="89900062-c475-4e1e-b926-49068af56754" containerName="maas-api" containerID="cri-o://addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41" gracePeriod=30 Sep 16 01:37:16.522172 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.522105 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:37:16.561978 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.561947 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-metrics-certs\") pod \"89900062-c475-4e1e-b926-49068af56754\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " Sep 16 01:37:16.562156 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.562057 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqzrn\" (UniqueName: \"kubernetes.io/projected/89900062-c475-4e1e-b926-49068af56754-kube-api-access-qqzrn\") pod \"89900062-c475-4e1e-b926-49068af56754\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " Sep 16 01:37:16.562156 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.562088 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-maas-api-tls\") pod \"89900062-c475-4e1e-b926-49068af56754\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " Sep 16 01:37:16.562156 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.562121 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89900062-c475-4e1e-b926-49068af56754-openshift-service-ca\") pod \"89900062-c475-4e1e-b926-49068af56754\" (UID: \"89900062-c475-4e1e-b926-49068af56754\") " Sep 16 01:37:16.562631 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.562597 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89900062-c475-4e1e-b926-49068af56754-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89900062-c475-4e1e-b926-49068af56754" (UID: "89900062-c475-4e1e-b926-49068af56754"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:37:16.564353 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.564312 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89900062-c475-4e1e-b926-49068af56754" (UID: "89900062-c475-4e1e-b926-49068af56754"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:37:16.564460 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.564410 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "89900062-c475-4e1e-b926-49068af56754" (UID: "89900062-c475-4e1e-b926-49068af56754"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:37:16.564509 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.564455 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89900062-c475-4e1e-b926-49068af56754-kube-api-access-qqzrn" (OuterVolumeSpecName: "kube-api-access-qqzrn") pod "89900062-c475-4e1e-b926-49068af56754" (UID: "89900062-c475-4e1e-b926-49068af56754"). InnerVolumeSpecName "kube-api-access-qqzrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:37:16.663887 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.663847 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:37:16.663887 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.663892 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqzrn\" (UniqueName: \"kubernetes.io/projected/89900062-c475-4e1e-b926-49068af56754-kube-api-access-qqzrn\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:37:16.664088 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.663908 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89900062-c475-4e1e-b926-49068af56754-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:37:16.664088 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:16.663922 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89900062-c475-4e1e-b926-49068af56754-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:37:17.187445 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:17.187410 2567 generic.go:356] "Generic (PLEG): container finished" podID="89900062-c475-4e1e-b926-49068af56754" containerID="addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41" exitCode=0 Sep 16 01:37:17.187640 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:17.187477 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" Sep 16 01:37:17.187640 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:17.187501 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" event={"ID":"89900062-c475-4e1e-b926-49068af56754","Type":"ContainerDied","Data":"addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41"} Sep 16 01:37:17.187640 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:17.187558 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt" event={"ID":"89900062-c475-4e1e-b926-49068af56754","Type":"ContainerDied","Data":"34da46488978ec9a7749af03cb2e0114caecef1feeb3993c2cc2efa3360ad606"} Sep 16 01:37:17.187640 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:17.187596 2567 scope.go:117] "RemoveContainer" containerID="addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41" Sep 16 01:37:17.198357 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:17.198331 2567 scope.go:117] "RemoveContainer" containerID="addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41" Sep 16 01:37:17.198737 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:37:17.198714 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41\": container with ID starting with addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41 not found: ID does not exist" containerID="addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41" Sep 16 01:37:17.198809 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:17.198747 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41"} err="failed to get container status \"addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41\": rpc error: code = NotFound desc = could not find container \"addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41\": container with ID starting with addd62cd65de8962bf1eaa39404845a83c15abfb0fc098574a554151ccec1c41 not found: ID does not exist" Sep 16 01:37:17.213108 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:17.213081 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt"] Sep 16 01:37:17.216568 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:17.216546 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-482177-7b44cc8f87-dfmpt"] Sep 16 01:37:17.939830 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:17.939798 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89900062-c475-4e1e-b926-49068af56754" path="/var/lib/kubelet/pods/89900062-c475-4e1e-b926-49068af56754/volumes" Sep 16 01:37:31.257032 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.257000 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74"] Sep 16 01:37:31.257638 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.257600 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89900062-c475-4e1e-b926-49068af56754" containerName="maas-api" Sep 16 01:37:31.257638 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.257618 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="89900062-c475-4e1e-b926-49068af56754" containerName="maas-api" Sep 16 01:37:31.257752 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.257739 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="89900062-c475-4e1e-b926-49068af56754" containerName="maas-api" Sep 16 01:37:31.262324 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.262301 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:31.264574 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.264553 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-61dff065-serving-cert\"" Sep 16 01:37:31.271109 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.271077 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74"] Sep 16 01:37:31.416289 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.416249 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2t7g\" (UniqueName: \"kubernetes.io/projected/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-kube-api-access-c2t7g\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:31.416467 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.416370 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-openshift-service-ca\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:31.416467 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.416458 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-maas-api-tls\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:31.416548 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.416514 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-metrics-certs\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:31.518000 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.517911 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-maas-api-tls\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:31.518000 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.517973 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-metrics-certs\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:31.518217 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.518018 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2t7g\" (UniqueName: \"kubernetes.io/projected/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-kube-api-access-c2t7g\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:31.518217 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.518063 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-openshift-service-ca\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:31.518217 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:37:31.518079 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-serving-cert: secret "maas-api-e2e-mt-61dff065-serving-cert" not found Sep 16 01:37:31.518217 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:37:31.518159 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-maas-api-tls podName:6f3fdd19-0182-42f3-95bb-ee7ac79990f5 nodeName:}" failed. No retries permitted until 2026-09-16 01:37:32.018137 +0000 UTC m=+1302.603939194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-maas-api-tls") pod "maas-api-e2e-mt-61dff065-95889c5c9-96z74" (UID: "6f3fdd19-0182-42f3-95bb-ee7ac79990f5") : secret "maas-api-e2e-mt-61dff065-serving-cert" not found Sep 16 01:37:31.518768 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.518741 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-openshift-service-ca\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:31.520787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.520764 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-metrics-certs\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:31.527187 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:31.527165 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2t7g\" (UniqueName: \"kubernetes.io/projected/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-kube-api-access-c2t7g\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:32.022956 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:32.022913 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-maas-api-tls\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:32.025553 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:32.025531 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-maas-api-tls\") pod \"maas-api-e2e-mt-61dff065-95889c5c9-96z74\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:32.178012 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:32.177971 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:32.449742 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:32.449714 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74"] Sep 16 01:37:32.451767 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:37:32.451739 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3fdd19_0182_42f3_95bb_ee7ac79990f5.slice/crio-087d884f4de330c3480ef40db54a46d6c0838f14fda3d9d99e28b8eae1fae49f WatchSource:0}: Error finding container 087d884f4de330c3480ef40db54a46d6c0838f14fda3d9d99e28b8eae1fae49f: Status 404 returned error can't find the container with id 087d884f4de330c3480ef40db54a46d6c0838f14fda3d9d99e28b8eae1fae49f Sep 16 01:37:33.274233 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:33.274193 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" event={"ID":"6f3fdd19-0182-42f3-95bb-ee7ac79990f5","Type":"ContainerStarted","Data":"d2478ebad6d1cd2aaddd13ff42646cb4cc3206752009299eb1df4630a1ff46b4"} Sep 16 01:37:33.274465 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:33.274247 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" event={"ID":"6f3fdd19-0182-42f3-95bb-ee7ac79990f5","Type":"ContainerStarted","Data":"087d884f4de330c3480ef40db54a46d6c0838f14fda3d9d99e28b8eae1fae49f"} Sep 16 01:37:33.274465 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:33.274297 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:37:33.289864 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:33.289811 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" podStartSLOduration=1.962965627 podStartE2EDuration="2.289794547s" totalImagesPullingTime="326.82892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:37:33.288963455 +0000 UTC m=+1303.874765668" watchObservedRunningTime="2026-09-16 01:37:33.289794547 +0000 UTC m=+1303.875596758" Sep 16 01:37:39.284875 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:37:39.284841 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:38:12.230930 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.230898 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74"] Sep 16 01:38:12.231475 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.231162 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" podUID="6f3fdd19-0182-42f3-95bb-ee7ac79990f5" containerName="maas-api" containerID="cri-o://d2478ebad6d1cd2aaddd13ff42646cb4cc3206752009299eb1df4630a1ff46b4" gracePeriod=30 Sep 16 01:38:12.475132 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.475097 2567 generic.go:356] "Generic (PLEG): container finished" podID="6f3fdd19-0182-42f3-95bb-ee7ac79990f5" containerID="d2478ebad6d1cd2aaddd13ff42646cb4cc3206752009299eb1df4630a1ff46b4" exitCode=0 Sep 16 01:38:12.475330 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.475165 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" event={"ID":"6f3fdd19-0182-42f3-95bb-ee7ac79990f5","Type":"ContainerDied","Data":"d2478ebad6d1cd2aaddd13ff42646cb4cc3206752009299eb1df4630a1ff46b4"} Sep 16 01:38:12.475330 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.475217 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" event={"ID":"6f3fdd19-0182-42f3-95bb-ee7ac79990f5","Type":"ContainerDied","Data":"087d884f4de330c3480ef40db54a46d6c0838f14fda3d9d99e28b8eae1fae49f"} Sep 16 01:38:12.475330 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.475233 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="087d884f4de330c3480ef40db54a46d6c0838f14fda3d9d99e28b8eae1fae49f" Sep 16 01:38:12.492811 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.492751 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:38:12.535702 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.535676 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-openshift-service-ca\") pod \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " Sep 16 01:38:12.535887 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.535731 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-metrics-certs\") pod \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " Sep 16 01:38:12.535887 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.535784 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2t7g\" (UniqueName: \"kubernetes.io/projected/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-kube-api-access-c2t7g\") pod \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " Sep 16 01:38:12.535887 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.535818 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-maas-api-tls\") pod \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\" (UID: \"6f3fdd19-0182-42f3-95bb-ee7ac79990f5\") " Sep 16 01:38:12.536413 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.536378 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f3fdd19-0182-42f3-95bb-ee7ac79990f5" (UID: "6f3fdd19-0182-42f3-95bb-ee7ac79990f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:38:12.538399 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.538374 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6f3fdd19-0182-42f3-95bb-ee7ac79990f5" (UID: "6f3fdd19-0182-42f3-95bb-ee7ac79990f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:38:12.538550 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.538519 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f3fdd19-0182-42f3-95bb-ee7ac79990f5" (UID: "6f3fdd19-0182-42f3-95bb-ee7ac79990f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:38:12.538550 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.538533 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-kube-api-access-c2t7g" (OuterVolumeSpecName: "kube-api-access-c2t7g") pod "6f3fdd19-0182-42f3-95bb-ee7ac79990f5" (UID: "6f3fdd19-0182-42f3-95bb-ee7ac79990f5"). InnerVolumeSpecName "kube-api-access-c2t7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:38:12.637161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.637133 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:12.637161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.637155 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2t7g\" (UniqueName: \"kubernetes.io/projected/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-kube-api-access-c2t7g\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:12.637161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.637166 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:12.637161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:12.637174 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f3fdd19-0182-42f3-95bb-ee7ac79990f5-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:13.481394 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:13.481359 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74" Sep 16 01:38:13.509779 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:13.509742 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74"] Sep 16 01:38:13.511593 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:13.511568 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61dff065-95889c5c9-96z74"] Sep 16 01:38:13.940386 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:13.940301 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f3fdd19-0182-42f3-95bb-ee7ac79990f5" path="/var/lib/kubelet/pods/6f3fdd19-0182-42f3-95bb-ee7ac79990f5/volumes" Sep 16 01:38:34.407314 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:34.407251 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv"] Sep 16 01:38:34.431090 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:34.431056 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9"] Sep 16 01:38:37.465859 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.465816 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf"] Sep 16 01:38:37.466443 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.466424 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f3fdd19-0182-42f3-95bb-ee7ac79990f5" containerName="maas-api" Sep 16 01:38:37.466537 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.466446 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f3fdd19-0182-42f3-95bb-ee7ac79990f5" containerName="maas-api" Sep 16 01:38:37.466596 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.466579 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f3fdd19-0182-42f3-95bb-ee7ac79990f5" containerName="maas-api" Sep 16 01:38:37.469916 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.469897 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:37.472421 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.472384 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-4bc0fb-serving-cert\"" Sep 16 01:38:37.480826 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.480597 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf"] Sep 16 01:38:37.588409 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.588362 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kckf\" (UniqueName: \"kubernetes.io/projected/b29099ae-0d49-4ec1-9441-2aec696cc971-kube-api-access-6kckf\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:37.588617 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.588428 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:37.588617 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.588457 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b29099ae-0d49-4ec1-9441-2aec696cc971-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:37.588617 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.588558 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:37.689272 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.689220 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:37.689492 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.689308 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b29099ae-0d49-4ec1-9441-2aec696cc971-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:37.689492 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:38:37.689384 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-serving-cert: secret "maas-api-e2e-shared-a-w1-4bc0fb-serving-cert" not found Sep 16 01:38:37.689492 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.689408 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:37.689492 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:38:37.689455 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-maas-api-tls podName:b29099ae-0d49-4ec1-9441-2aec696cc971 nodeName:}" failed. No retries permitted until 2026-09-16 01:38:38.189439423 +0000 UTC m=+1368.775241612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-maas-api-tls") pod "maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" (UID: "b29099ae-0d49-4ec1-9441-2aec696cc971") : secret "maas-api-e2e-shared-a-w1-4bc0fb-serving-cert" not found Sep 16 01:38:37.689721 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.689548 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kckf\" (UniqueName: \"kubernetes.io/projected/b29099ae-0d49-4ec1-9441-2aec696cc971-kube-api-access-6kckf\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:37.690099 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.690074 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b29099ae-0d49-4ec1-9441-2aec696cc971-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:37.692321 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.692303 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:37.699609 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:37.699577 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kckf\" (UniqueName: \"kubernetes.io/projected/b29099ae-0d49-4ec1-9441-2aec696cc971-kube-api-access-6kckf\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:38.195553 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:38.195518 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:38.198226 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:38.198192 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:38.382859 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:38.382805 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:38.656565 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:38.656536 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf"] Sep 16 01:38:38.658691 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:38:38.658662 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29099ae_0d49_4ec1_9441_2aec696cc971.slice/crio-54c63609614ecb4c617fbb1b1b4c0d8689d60f6f8eba9f16c26445bc01992e3d WatchSource:0}: Error finding container 54c63609614ecb4c617fbb1b1b4c0d8689d60f6f8eba9f16c26445bc01992e3d: Status 404 returned error can't find the container with id 54c63609614ecb4c617fbb1b1b4c0d8689d60f6f8eba9f16c26445bc01992e3d Sep 16 01:38:38.660053 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:38.660033 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 01:38:39.613757 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:39.613723 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" event={"ID":"b29099ae-0d49-4ec1-9441-2aec696cc971","Type":"ContainerStarted","Data":"5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d"} Sep 16 01:38:39.613944 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:39.613786 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" event={"ID":"b29099ae-0d49-4ec1-9441-2aec696cc971","Type":"ContainerStarted","Data":"54c63609614ecb4c617fbb1b1b4c0d8689d60f6f8eba9f16c26445bc01992e3d"} Sep 16 01:38:39.613944 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:39.613902 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:39.629642 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:39.629586 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" podStartSLOduration=2.158730348 podStartE2EDuration="2.629564342s" totalImagesPullingTime="470.833994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:38:39.628477334 +0000 UTC m=+1370.214279546" watchObservedRunningTime="2026-09-16 01:38:39.629564342 +0000 UTC m=+1370.215366553" Sep 16 01:38:45.383552 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.383517 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr"] Sep 16 01:38:45.384072 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.383767 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" podUID="a9471daf-6d5c-4dff-92fb-6d8b215cda34" containerName="maas-api" containerID="cri-o://9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26" gracePeriod=30 Sep 16 01:38:45.624544 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.624518 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:38:45.642161 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.642080 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:38:45.646736 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.646707 2567 generic.go:356] "Generic (PLEG): container finished" podID="a9471daf-6d5c-4dff-92fb-6d8b215cda34" containerID="9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26" exitCode=0 Sep 16 01:38:45.646876 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.646748 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" Sep 16 01:38:45.646876 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.646797 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" event={"ID":"a9471daf-6d5c-4dff-92fb-6d8b215cda34","Type":"ContainerDied","Data":"9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26"} Sep 16 01:38:45.646876 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.646855 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr" event={"ID":"a9471daf-6d5c-4dff-92fb-6d8b215cda34","Type":"ContainerDied","Data":"42bf8d81c80ff369fdfde1e80d7f739568d12858dd036dfc4293d3b32b0017a0"} Sep 16 01:38:45.646994 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.646881 2567 scope.go:117] "RemoveContainer" containerID="9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26" Sep 16 01:38:45.659088 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.658977 2567 scope.go:117] "RemoveContainer" containerID="9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26" Sep 16 01:38:45.660167 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:38:45.660011 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26\": container with ID starting with 9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26 not found: ID does not exist" containerID="9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26" Sep 16 01:38:45.660167 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.660050 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26"} err="failed to get container status \"9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26\": rpc error: code = NotFound desc = could not find container \"9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26\": container with ID starting with 9ba2fb5e88a10ee4241c37508d54dc0dd6d6ac56a87bf0ff1cc7b53b59bfaa26 not found: ID does not exist" Sep 16 01:38:45.667382 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.667357 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxhfl\" (UniqueName: \"kubernetes.io/projected/a9471daf-6d5c-4dff-92fb-6d8b215cda34-kube-api-access-bxhfl\") pod \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " Sep 16 01:38:45.667549 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.667405 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9471daf-6d5c-4dff-92fb-6d8b215cda34-openshift-service-ca\") pod \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " Sep 16 01:38:45.667549 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.667464 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-maas-api-tls\") pod \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " Sep 16 01:38:45.667549 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.667544 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-metrics-certs\") pod \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\" (UID: \"a9471daf-6d5c-4dff-92fb-6d8b215cda34\") " Sep 16 01:38:45.667878 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.667849 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9471daf-6d5c-4dff-92fb-6d8b215cda34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a9471daf-6d5c-4dff-92fb-6d8b215cda34" (UID: "a9471daf-6d5c-4dff-92fb-6d8b215cda34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:38:45.668556 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.668525 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9471daf-6d5c-4dff-92fb-6d8b215cda34-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:45.669941 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.669915 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a9471daf-6d5c-4dff-92fb-6d8b215cda34" (UID: "a9471daf-6d5c-4dff-92fb-6d8b215cda34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:38:45.669941 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.669932 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9471daf-6d5c-4dff-92fb-6d8b215cda34-kube-api-access-bxhfl" (OuterVolumeSpecName: "kube-api-access-bxhfl") pod "a9471daf-6d5c-4dff-92fb-6d8b215cda34" (UID: "a9471daf-6d5c-4dff-92fb-6d8b215cda34"). InnerVolumeSpecName "kube-api-access-bxhfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:38:45.670364 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.670337 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a9471daf-6d5c-4dff-92fb-6d8b215cda34" (UID: "a9471daf-6d5c-4dff-92fb-6d8b215cda34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:38:45.769756 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.769725 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:45.769756 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.769751 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9471daf-6d5c-4dff-92fb-6d8b215cda34-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:45.769756 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.769760 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxhfl\" (UniqueName: \"kubernetes.io/projected/a9471daf-6d5c-4dff-92fb-6d8b215cda34-kube-api-access-bxhfl\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:45.961711 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.961674 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr"] Sep 16 01:38:45.965265 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:45.965232 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d0c9f2-56bb97b9cf-js7qr"] Sep 16 01:38:47.939595 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:47.939558 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9471daf-6d5c-4dff-92fb-6d8b215cda34" path="/var/lib/kubelet/pods/a9471daf-6d5c-4dff-92fb-6d8b215cda34/volumes" Sep 16 01:38:49.427238 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.427193 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" podUID="13839e18-47af-455c-9aba-6be1cefc75cd" containerName="main" containerID="cri-o://8e5a701b7075569dd9e607d586412e23f883e9ef17e20c245dd96b9788f74e01" gracePeriod=45 Sep 16 01:38:49.450518 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.450477 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" podUID="f120328d-630b-40ce-8d1a-8124ca1f3c1c" containerName="main" containerID="cri-o://2be2d65b926d5a0d43b22db0a2960f4bdbce6ff946d75861ae17924d4ebecfce" gracePeriod=45 Sep 16 01:38:49.672004 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.671972 2567 generic.go:356] "Generic (PLEG): container finished" podID="f120328d-630b-40ce-8d1a-8124ca1f3c1c" containerID="2be2d65b926d5a0d43b22db0a2960f4bdbce6ff946d75861ae17924d4ebecfce" exitCode=0 Sep 16 01:38:49.672187 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.672041 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" event={"ID":"f120328d-630b-40ce-8d1a-8124ca1f3c1c","Type":"ContainerDied","Data":"2be2d65b926d5a0d43b22db0a2960f4bdbce6ff946d75861ae17924d4ebecfce"} Sep 16 01:38:49.674115 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.674090 2567 generic.go:356] "Generic (PLEG): container finished" podID="13839e18-47af-455c-9aba-6be1cefc75cd" containerID="8e5a701b7075569dd9e607d586412e23f883e9ef17e20c245dd96b9788f74e01" exitCode=0 Sep 16 01:38:49.674267 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.674180 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" event={"ID":"13839e18-47af-455c-9aba-6be1cefc75cd","Type":"ContainerDied","Data":"8e5a701b7075569dd9e607d586412e23f883e9ef17e20c245dd96b9788f74e01"} Sep 16 01:38:49.731545 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.731503 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:38:49.734585 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.734558 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:38:49.810549 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.810522 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcj62\" (UniqueName: \"kubernetes.io/projected/13839e18-47af-455c-9aba-6be1cefc75cd-kube-api-access-dcj62\") pod \"13839e18-47af-455c-9aba-6be1cefc75cd\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " Sep 16 01:38:49.810549 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.810553 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwldl\" (UniqueName: \"kubernetes.io/projected/f120328d-630b-40ce-8d1a-8124ca1f3c1c-kube-api-access-mwldl\") pod \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " Sep 16 01:38:49.810779 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.810592 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tls-certs\") pod \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " Sep 16 01:38:49.810779 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.810642 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-dshm\") pod \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " Sep 16 01:38:49.810779 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.810684 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-model-cache\") pod \"13839e18-47af-455c-9aba-6be1cefc75cd\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " Sep 16 01:38:49.810779 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.810740 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-home\") pod \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " Sep 16 01:38:49.810779 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.810761 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-model-cache\") pod \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " Sep 16 01:38:49.811036 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.810784 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-dshm\") pod \"13839e18-47af-455c-9aba-6be1cefc75cd\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " Sep 16 01:38:49.811036 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.810821 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13839e18-47af-455c-9aba-6be1cefc75cd-tls-certs\") pod \"13839e18-47af-455c-9aba-6be1cefc75cd\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " Sep 16 01:38:49.811036 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.810851 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tmp-dir\") pod \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\" (UID: \"f120328d-630b-40ce-8d1a-8124ca1f3c1c\") " Sep 16 01:38:49.811036 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.810908 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-home\") pod \"13839e18-47af-455c-9aba-6be1cefc75cd\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " Sep 16 01:38:49.811036 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.811010 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-tmp-dir\") pod \"13839e18-47af-455c-9aba-6be1cefc75cd\" (UID: \"13839e18-47af-455c-9aba-6be1cefc75cd\") " Sep 16 01:38:49.811316 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.811177 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "13839e18-47af-455c-9aba-6be1cefc75cd" (UID: "13839e18-47af-455c-9aba-6be1cefc75cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.812170 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.811437 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-model-cache\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.812170 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.811564 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-home" (OuterVolumeSpecName: "home") pod "f120328d-630b-40ce-8d1a-8124ca1f3c1c" (UID: "f120328d-630b-40ce-8d1a-8124ca1f3c1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.812170 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.811579 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13839e18-47af-455c-9aba-6be1cefc75cd" (UID: "13839e18-47af-455c-9aba-6be1cefc75cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.812170 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.811733 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f120328d-630b-40ce-8d1a-8124ca1f3c1c" (UID: "f120328d-630b-40ce-8d1a-8124ca1f3c1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.812170 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.811798 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-home" (OuterVolumeSpecName: "home") pod "13839e18-47af-455c-9aba-6be1cefc75cd" (UID: "13839e18-47af-455c-9aba-6be1cefc75cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.812170 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.812094 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f120328d-630b-40ce-8d1a-8124ca1f3c1c" (UID: "f120328d-630b-40ce-8d1a-8124ca1f3c1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.813839 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.813778 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13839e18-47af-455c-9aba-6be1cefc75cd-kube-api-access-dcj62" (OuterVolumeSpecName: "kube-api-access-dcj62") pod "13839e18-47af-455c-9aba-6be1cefc75cd" (UID: "13839e18-47af-455c-9aba-6be1cefc75cd"). InnerVolumeSpecName "kube-api-access-dcj62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:38:49.813839 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.813788 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f120328d-630b-40ce-8d1a-8124ca1f3c1c" (UID: "f120328d-630b-40ce-8d1a-8124ca1f3c1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:38:49.813839 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.813779 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f120328d-630b-40ce-8d1a-8124ca1f3c1c-kube-api-access-mwldl" (OuterVolumeSpecName: "kube-api-access-mwldl") pod "f120328d-630b-40ce-8d1a-8124ca1f3c1c" (UID: "f120328d-630b-40ce-8d1a-8124ca1f3c1c"). InnerVolumeSpecName "kube-api-access-mwldl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:38:49.813839 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.813823 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-dshm" (OuterVolumeSpecName: "dshm") pod "f120328d-630b-40ce-8d1a-8124ca1f3c1c" (UID: "f120328d-630b-40ce-8d1a-8124ca1f3c1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.814044 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.813958 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-dshm" (OuterVolumeSpecName: "dshm") pod "13839e18-47af-455c-9aba-6be1cefc75cd" (UID: "13839e18-47af-455c-9aba-6be1cefc75cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:38:49.814082 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.814054 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13839e18-47af-455c-9aba-6be1cefc75cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13839e18-47af-455c-9aba-6be1cefc75cd" (UID: "13839e18-47af-455c-9aba-6be1cefc75cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:38:49.912317 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.912264 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tls-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.912317 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.912314 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-dshm\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.912317 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.912323 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-home\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.912632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.912331 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-model-cache\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.912632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.912345 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-dshm\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.912632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.912353 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13839e18-47af-455c-9aba-6be1cefc75cd-tls-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.912632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.912360 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f120328d-630b-40ce-8d1a-8124ca1f3c1c-tmp-dir\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.912632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.912367 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-home\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.912632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.912374 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13839e18-47af-455c-9aba-6be1cefc75cd-tmp-dir\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.912632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.912382 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcj62\" (UniqueName: \"kubernetes.io/projected/13839e18-47af-455c-9aba-6be1cefc75cd-kube-api-access-dcj62\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:49.912632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:49.912390 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwldl\" (UniqueName: \"kubernetes.io/projected/f120328d-630b-40ce-8d1a-8124ca1f3c1c-kube-api-access-mwldl\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:38:50.043812 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:50.043778 2567 scope.go:117] "RemoveContainer" containerID="2be2d65b926d5a0d43b22db0a2960f4bdbce6ff946d75861ae17924d4ebecfce" Sep 16 01:38:50.680494 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:50.680453 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" event={"ID":"f120328d-630b-40ce-8d1a-8124ca1f3c1c","Type":"ContainerDied","Data":"f34128566bc5be9042c30b36dd5cd5d69759647650749b2120feed66968c796f"} Sep 16 01:38:50.682251 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:50.682220 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" Sep 16 01:38:50.682251 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:50.682232 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv" event={"ID":"13839e18-47af-455c-9aba-6be1cefc75cd","Type":"ContainerDied","Data":"ddb7fdcc6b1f8022c64c156a123dbeaa66d999ef73f78526b3cf5c8547dd63e8"} Sep 16 01:38:50.682488 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:50.682296 2567 scope.go:117] "RemoveContainer" containerID="8e5a701b7075569dd9e607d586412e23f883e9ef17e20c245dd96b9788f74e01" Sep 16 01:38:50.682488 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:50.682225 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9" Sep 16 01:38:50.717509 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:50.717476 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv"] Sep 16 01:38:50.720777 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:50.720746 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/distinct-2-w3-d0c9f2-kserve-d555654cf-vbrmv"] Sep 16 01:38:50.730681 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:50.730651 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9"] Sep 16 01:38:50.735955 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:50.735929 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d0c9f2/simulator-w3-d0c9f2-kserve-5b496cffd4-985b9"] Sep 16 01:38:51.939744 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:51.939712 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13839e18-47af-455c-9aba-6be1cefc75cd" path="/var/lib/kubelet/pods/13839e18-47af-455c-9aba-6be1cefc75cd/volumes" Sep 16 01:38:51.940112 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:51.940099 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f120328d-630b-40ce-8d1a-8124ca1f3c1c" path="/var/lib/kubelet/pods/f120328d-630b-40ce-8d1a-8124ca1f3c1c/volumes" Sep 16 01:38:55.560135 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.560102 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s"] Sep 16 01:38:55.560755 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.560737 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f120328d-630b-40ce-8d1a-8124ca1f3c1c" containerName="main" Sep 16 01:38:55.560826 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.560759 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f120328d-630b-40ce-8d1a-8124ca1f3c1c" containerName="main" Sep 16 01:38:55.560826 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.560771 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13839e18-47af-455c-9aba-6be1cefc75cd" containerName="main" Sep 16 01:38:55.560826 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.560780 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="13839e18-47af-455c-9aba-6be1cefc75cd" containerName="main" Sep 16 01:38:55.560826 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.560794 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9471daf-6d5c-4dff-92fb-6d8b215cda34" containerName="maas-api" Sep 16 01:38:55.560826 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.560803 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9471daf-6d5c-4dff-92fb-6d8b215cda34" containerName="maas-api" Sep 16 01:38:55.561079 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.560907 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f120328d-630b-40ce-8d1a-8124ca1f3c1c" containerName="main" Sep 16 01:38:55.561079 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.560923 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="13839e18-47af-455c-9aba-6be1cefc75cd" containerName="main" Sep 16 01:38:55.561079 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.560935 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9471daf-6d5c-4dff-92fb-6d8b215cda34" containerName="maas-api" Sep 16 01:38:55.566300 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.566256 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:55.568850 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.568821 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-0d2655-serving-cert\"" Sep 16 01:38:55.571882 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.571709 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s"] Sep 16 01:38:55.671447 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.671405 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c32416f-e4bd-4be7-9378-5561418569c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:55.671649 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.671468 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:55.671649 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.671542 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29jcm\" (UniqueName: \"kubernetes.io/projected/1c32416f-e4bd-4be7-9378-5561418569c9-kube-api-access-29jcm\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:55.671649 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.671604 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:55.773147 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.773102 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c32416f-e4bd-4be7-9378-5561418569c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:55.773321 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.773177 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:55.773321 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.773242 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29jcm\" (UniqueName: \"kubernetes.io/projected/1c32416f-e4bd-4be7-9378-5561418569c9-kube-api-access-29jcm\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:55.773440 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:38:55.773339 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-serving-cert: secret "maas-api-e2e-shared-b-w1-0d2655-serving-cert" not found Sep 16 01:38:55.773440 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:38:55.773417 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-maas-api-tls podName:1c32416f-e4bd-4be7-9378-5561418569c9 nodeName:}" failed. No retries permitted until 2026-09-16 01:38:56.273394448 +0000 UTC m=+1386.859196641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-maas-api-tls") pod "maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" (UID: "1c32416f-e4bd-4be7-9378-5561418569c9") : secret "maas-api-e2e-shared-b-w1-0d2655-serving-cert" not found Sep 16 01:38:55.773440 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.773344 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:55.773859 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.773838 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c32416f-e4bd-4be7-9378-5561418569c9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:55.776121 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.776088 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:55.783437 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:55.783410 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29jcm\" (UniqueName: \"kubernetes.io/projected/1c32416f-e4bd-4be7-9378-5561418569c9-kube-api-access-29jcm\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:56.279693 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:56.279660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:56.282449 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:56.282420 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:56.480940 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:56.480907 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:56.752449 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:56.752420 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s"] Sep 16 01:38:56.754258 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:38:56.754233 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c32416f_e4bd_4be7_9378_5561418569c9.slice/crio-59fc70db24cdf4325a162c5947782742014bb8a9be41ceeb4813bb2bb8454352 WatchSource:0}: Error finding container 59fc70db24cdf4325a162c5947782742014bb8a9be41ceeb4813bb2bb8454352: Status 404 returned error can't find the container with id 59fc70db24cdf4325a162c5947782742014bb8a9be41ceeb4813bb2bb8454352 Sep 16 01:38:57.720893 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:57.720854 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" event={"ID":"1c32416f-e4bd-4be7-9378-5561418569c9","Type":"ContainerStarted","Data":"adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f"} Sep 16 01:38:57.721110 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:57.720912 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" event={"ID":"1c32416f-e4bd-4be7-9378-5561418569c9","Type":"ContainerStarted","Data":"59fc70db24cdf4325a162c5947782742014bb8a9be41ceeb4813bb2bb8454352"} Sep 16 01:38:57.721110 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:57.720954 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:38:57.736593 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:38:57.736537 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" podStartSLOduration=2.368092225 podStartE2EDuration="2.736518025s" totalImagesPullingTime="368.4258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:38:57.736013049 +0000 UTC m=+1388.321815261" watchObservedRunningTime="2026-09-16 01:38:57.736518025 +0000 UTC m=+1388.322320235" Sep 16 01:39:03.731149 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:03.731120 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:39:22.040718 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:22.040682 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9135-898bc3"] Sep 16 01:39:22.043769 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:22.043751 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-898bc3" Sep 16 01:39:22.046222 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:22.046195 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6n2fb\"" Sep 16 01:39:22.047592 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:22.047565 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9135-898bc3"] Sep 16 01:39:22.133465 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:22.133426 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l72s4\" (UniqueName: \"kubernetes.io/projected/1004678d-9175-4909-9d22-0fd1c4204799-kube-api-access-l72s4\") pod \"test-curl-9135-898bc3\" (UID: \"1004678d-9175-4909-9d22-0fd1c4204799\") " pod="opendatahub/test-curl-9135-898bc3" Sep 16 01:39:22.234375 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:22.234334 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l72s4\" (UniqueName: \"kubernetes.io/projected/1004678d-9175-4909-9d22-0fd1c4204799-kube-api-access-l72s4\") pod \"test-curl-9135-898bc3\" (UID: \"1004678d-9175-4909-9d22-0fd1c4204799\") " pod="opendatahub/test-curl-9135-898bc3" Sep 16 01:39:22.242374 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:22.242339 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l72s4\" (UniqueName: \"kubernetes.io/projected/1004678d-9175-4909-9d22-0fd1c4204799-kube-api-access-l72s4\") pod \"test-curl-9135-898bc3\" (UID: \"1004678d-9175-4909-9d22-0fd1c4204799\") " pod="opendatahub/test-curl-9135-898bc3" Sep 16 01:39:22.354747 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:22.354666 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-898bc3" Sep 16 01:39:22.622482 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:22.622456 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9135-898bc3"] Sep 16 01:39:22.624425 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:39:22.624395 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1004678d_9175_4909_9d22_0fd1c4204799.slice/crio-a6aff972438e600625da3daaab3ac5af582817a2a1cfd35ebf868b6d4b46edde WatchSource:0}: Error finding container a6aff972438e600625da3daaab3ac5af582817a2a1cfd35ebf868b6d4b46edde: Status 404 returned error can't find the container with id a6aff972438e600625da3daaab3ac5af582817a2a1cfd35ebf868b6d4b46edde Sep 16 01:39:22.850848 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:22.850814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9135-898bc3" event={"ID":"1004678d-9175-4909-9d22-0fd1c4204799","Type":"ContainerStarted","Data":"a6aff972438e600625da3daaab3ac5af582817a2a1cfd35ebf868b6d4b46edde"} Sep 16 01:39:24.862409 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:24.862372 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9135-898bc3" event={"ID":"1004678d-9175-4909-9d22-0fd1c4204799","Type":"ContainerStarted","Data":"760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57"} Sep 16 01:39:24.875603 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:24.875556 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9135-898bc3" podStartSLOduration=0.785259272 podStartE2EDuration="2.875537507s" totalImagesPullingTime="2.090278235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:39:24.87535153 +0000 UTC m=+1415.461153742" watchObservedRunningTime="2026-09-16 01:39:24.875537507 +0000 UTC m=+1415.461339720" Sep 16 01:39:25.484049 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:25.484011 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9135-898bc3"] Sep 16 01:39:25.487796 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:25.487771 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9135-898bc3"] Sep 16 01:39:25.867755 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:39:25.867682 2567 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9135-898bc3" container="test-curl-9135-898bc3" Sep 16 01:39:26.872860 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:26.872816 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9135-898bc3" podUID="1004678d-9175-4909-9d22-0fd1c4204799" containerName="test-curl-9135-898bc3" containerID="cri-o://760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57" gracePeriod=2 Sep 16 01:39:29.057765 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:29.057736 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-898bc3" Sep 16 01:39:29.102633 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:29.102600 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l72s4\" (UniqueName: \"kubernetes.io/projected/1004678d-9175-4909-9d22-0fd1c4204799-kube-api-access-l72s4\") pod \"1004678d-9175-4909-9d22-0fd1c4204799\" (UID: \"1004678d-9175-4909-9d22-0fd1c4204799\") " Sep 16 01:39:29.104954 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:29.104924 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1004678d-9175-4909-9d22-0fd1c4204799-kube-api-access-l72s4" (OuterVolumeSpecName: "kube-api-access-l72s4") pod "1004678d-9175-4909-9d22-0fd1c4204799" (UID: "1004678d-9175-4909-9d22-0fd1c4204799"). InnerVolumeSpecName "kube-api-access-l72s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:39:29.203851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:29.203772 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l72s4\" (UniqueName: \"kubernetes.io/projected/1004678d-9175-4909-9d22-0fd1c4204799-kube-api-access-l72s4\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:39:29.889263 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:29.889226 2567 generic.go:356] "Generic (PLEG): container finished" podID="1004678d-9175-4909-9d22-0fd1c4204799" containerID="760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57" exitCode=137 Sep 16 01:39:29.889457 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:29.889271 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9135-898bc3" Sep 16 01:39:29.889457 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:29.889333 2567 scope.go:117] "RemoveContainer" containerID="760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57" Sep 16 01:39:29.898267 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:29.898249 2567 scope.go:117] "RemoveContainer" containerID="760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57" Sep 16 01:39:29.898570 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:39:29.898546 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57\": container with ID starting with 760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57 not found: ID does not exist" containerID="760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57" Sep 16 01:39:29.898671 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:29.898577 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57"} err="failed to get container status \"760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57\": rpc error: code = NotFound desc = could not find container \"760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57\": container with ID starting with 760204e830563b04701755b5cf537e01e679f63dc6c55289750e6a8f20742d57 not found: ID does not exist" Sep 16 01:39:29.900714 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:39:29.900692 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9135-898bc3\" is forbidden: User \"system:node:ip-10-0-142-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-56.ec2.internal' and this object" podUID="1004678d-9175-4909-9d22-0fd1c4204799" pod="opendatahub/test-curl-9135-898bc3" Sep 16 01:39:29.940064 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:39:29.940022 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9135-898bc3\" is forbidden: User \"system:node:ip-10-0-142-56.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-56.ec2.internal' and this object" podUID="1004678d-9175-4909-9d22-0fd1c4204799" pod="opendatahub/test-curl-9135-898bc3" Sep 16 01:39:29.942551 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:29.942526 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1004678d-9175-4909-9d22-0fd1c4204799" path="/var/lib/kubelet/pods/1004678d-9175-4909-9d22-0fd1c4204799/volumes" Sep 16 01:39:35.540546 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.540513 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn"] Sep 16 01:39:35.541172 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.541151 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1004678d-9175-4909-9d22-0fd1c4204799" containerName="test-curl-9135-898bc3" Sep 16 01:39:35.541313 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.541175 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1004678d-9175-4909-9d22-0fd1c4204799" containerName="test-curl-9135-898bc3" Sep 16 01:39:35.541313 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.541261 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1004678d-9175-4909-9d22-0fd1c4204799" containerName="test-curl-9135-898bc3" Sep 16 01:39:35.546492 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.546468 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.549121 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.549099 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3b6e6489-serving-cert\"" Sep 16 01:39:35.555591 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.555567 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn"] Sep 16 01:39:35.672147 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.672107 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-maas-api-tls\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.672147 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.672156 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-metrics-certs\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.672414 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.672183 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b28645e-88c8-4adc-8269-4258dc3fe7e0-openshift-service-ca\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.672414 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.672338 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q6km\" (UniqueName: \"kubernetes.io/projected/9b28645e-88c8-4adc-8269-4258dc3fe7e0-kube-api-access-8q6km\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.773653 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.773617 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-maas-api-tls\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.773653 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.773658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-metrics-certs\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.773851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.773682 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b28645e-88c8-4adc-8269-4258dc3fe7e0-openshift-service-ca\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.773851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.773731 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q6km\" (UniqueName: \"kubernetes.io/projected/9b28645e-88c8-4adc-8269-4258dc3fe7e0-kube-api-access-8q6km\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.774465 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.774433 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b28645e-88c8-4adc-8269-4258dc3fe7e0-openshift-service-ca\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.776389 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.776354 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-maas-api-tls\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.776529 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.776507 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-metrics-certs\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.781242 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.781215 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q6km\" (UniqueName: \"kubernetes.io/projected/9b28645e-88c8-4adc-8269-4258dc3fe7e0-kube-api-access-8q6km\") pod \"maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:35.860350 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:35.860250 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:36.138595 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:36.138570 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn"] Sep 16 01:39:36.141081 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:39:36.141047 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b28645e_88c8_4adc_8269_4258dc3fe7e0.slice/crio-12e64963a2cfa3ed6f6a0c0fe95988ef856ac8e7197c46e02ed8aec5a5fcc857 WatchSource:0}: Error finding container 12e64963a2cfa3ed6f6a0c0fe95988ef856ac8e7197c46e02ed8aec5a5fcc857: Status 404 returned error can't find the container with id 12e64963a2cfa3ed6f6a0c0fe95988ef856ac8e7197c46e02ed8aec5a5fcc857 Sep 16 01:39:36.928853 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:36.928763 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" event={"ID":"9b28645e-88c8-4adc-8269-4258dc3fe7e0","Type":"ContainerStarted","Data":"9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260"} Sep 16 01:39:36.928853 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:36.928820 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" event={"ID":"9b28645e-88c8-4adc-8269-4258dc3fe7e0","Type":"ContainerStarted","Data":"12e64963a2cfa3ed6f6a0c0fe95988ef856ac8e7197c46e02ed8aec5a5fcc857"} Sep 16 01:39:36.928853 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:36.928842 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:39:36.945389 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:36.945344 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" podStartSLOduration=1.563614314 podStartE2EDuration="1.945329016s" totalImagesPullingTime="381.714702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:39:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:39:36.943525309 +0000 UTC m=+1427.529327520" watchObservedRunningTime="2026-09-16 01:39:36.945329016 +0000 UTC m=+1427.531131226" Sep 16 01:39:42.940730 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:39:42.940658 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:40:26.529455 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:26.529413 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9"] Sep 16 01:40:34.012205 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.012170 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7"] Sep 16 01:40:34.015872 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.015854 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.018387 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.018368 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-cbb499\"/\"sub-test-model-cbb499-kserve-self-signed-certs\"" Sep 16 01:40:34.025474 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.025454 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7"] Sep 16 01:40:34.044894 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.044858 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3453bb-6596-456e-87cb-c742cbe85d53-tls-certs\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.045049 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.044929 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcgdv\" (UniqueName: \"kubernetes.io/projected/7d3453bb-6596-456e-87cb-c742cbe85d53-kube-api-access-xcgdv\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.045049 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.044955 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-tmp-dir\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.045049 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.045028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-model-cache\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.045231 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.045084 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-dshm\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.045231 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.045172 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-home\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.146378 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.146341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-home\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.146569 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.146389 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3453bb-6596-456e-87cb-c742cbe85d53-tls-certs\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.146569 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.146452 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcgdv\" (UniqueName: \"kubernetes.io/projected/7d3453bb-6596-456e-87cb-c742cbe85d53-kube-api-access-xcgdv\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.146569 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.146478 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-tmp-dir\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.146569 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.146516 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-model-cache\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.146569 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.146566 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-dshm\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.146852 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.146833 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-home\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.146904 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.146847 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-tmp-dir\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.146966 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.146944 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-model-cache\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.148973 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.148952 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-dshm\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.149143 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.149126 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3453bb-6596-456e-87cb-c742cbe85d53-tls-certs\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.153822 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.153796 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcgdv\" (UniqueName: \"kubernetes.io/projected/7d3453bb-6596-456e-87cb-c742cbe85d53-kube-api-access-xcgdv\") pod \"sub-test-model-cbb499-kserve-7dd57f5c65-87wv7\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.327206 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.327114 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:34.601774 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:34.601701 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7"] Sep 16 01:40:34.605248 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:40:34.605212 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d3453bb_6596_456e_87cb_c742cbe85d53.slice/crio-69b56bbd083e0fc258205068c919208de7374fd844bcc5d3c25c53f290776f91 WatchSource:0}: Error finding container 69b56bbd083e0fc258205068c919208de7374fd844bcc5d3c25c53f290776f91: Status 404 returned error can't find the container with id 69b56bbd083e0fc258205068c919208de7374fd844bcc5d3c25c53f290776f91 Sep 16 01:40:35.229563 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:35.229521 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" event={"ID":"7d3453bb-6596-456e-87cb-c742cbe85d53","Type":"ContainerStarted","Data":"43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202"} Sep 16 01:40:35.229952 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:35.229574 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" event={"ID":"7d3453bb-6596-456e-87cb-c742cbe85d53","Type":"ContainerStarted","Data":"69b56bbd083e0fc258205068c919208de7374fd844bcc5d3c25c53f290776f91"} Sep 16 01:40:35.248293 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:35.248228 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" podStartSLOduration=2.248213385 podStartE2EDuration="2.248213385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:40:35.245681178 +0000 UTC m=+1485.831483390" watchObservedRunningTime="2026-09-16 01:40:35.248213385 +0000 UTC m=+1485.834015865" Sep 16 01:40:41.549543 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.549487 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" podUID="ddf96e2d-efcd-4be9-86f2-4219f32ebb39" containerName="main" containerID="cri-o://b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e" gracePeriod=45 Sep 16 01:40:41.808074 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.808011 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:40:41.924744 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.924708 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tmp-dir\") pod \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " Sep 16 01:40:41.924744 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.924751 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-home\") pod \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " Sep 16 01:40:41.925005 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.924768 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tls-certs\") pod \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " Sep 16 01:40:41.925005 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.924788 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-dshm\") pod \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " Sep 16 01:40:41.925005 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.924810 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wxmm\" (UniqueName: \"kubernetes.io/projected/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-kube-api-access-2wxmm\") pod \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " Sep 16 01:40:41.925005 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.924878 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-model-cache\") pod \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\" (UID: \"ddf96e2d-efcd-4be9-86f2-4219f32ebb39\") " Sep 16 01:40:41.925194 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.925076 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddf96e2d-efcd-4be9-86f2-4219f32ebb39" (UID: "ddf96e2d-efcd-4be9-86f2-4219f32ebb39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:40:41.925194 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.925091 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-home" (OuterVolumeSpecName: "home") pod "ddf96e2d-efcd-4be9-86f2-4219f32ebb39" (UID: "ddf96e2d-efcd-4be9-86f2-4219f32ebb39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:40:41.925336 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.925232 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tmp-dir\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:40:41.925336 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.925252 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-home\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:40:41.925336 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.925258 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddf96e2d-efcd-4be9-86f2-4219f32ebb39" (UID: "ddf96e2d-efcd-4be9-86f2-4219f32ebb39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:40:41.927422 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.927392 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-dshm" (OuterVolumeSpecName: "dshm") pod "ddf96e2d-efcd-4be9-86f2-4219f32ebb39" (UID: "ddf96e2d-efcd-4be9-86f2-4219f32ebb39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:40:41.927422 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.927405 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddf96e2d-efcd-4be9-86f2-4219f32ebb39" (UID: "ddf96e2d-efcd-4be9-86f2-4219f32ebb39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:40:41.927599 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:41.927472 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-kube-api-access-2wxmm" (OuterVolumeSpecName: "kube-api-access-2wxmm") pod "ddf96e2d-efcd-4be9-86f2-4219f32ebb39" (UID: "ddf96e2d-efcd-4be9-86f2-4219f32ebb39"). InnerVolumeSpecName "kube-api-access-2wxmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:40:42.026649 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.026599 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-model-cache\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:40:42.026649 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.026633 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-tls-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:40:42.026649 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.026642 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-dshm\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:40:42.026649 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.026650 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wxmm\" (UniqueName: \"kubernetes.io/projected/ddf96e2d-efcd-4be9-86f2-4219f32ebb39-kube-api-access-2wxmm\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:40:42.266460 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.266423 2567 generic.go:356] "Generic (PLEG): container finished" podID="ddf96e2d-efcd-4be9-86f2-4219f32ebb39" containerID="b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e" exitCode=0 Sep 16 01:40:42.266661 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.266492 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" Sep 16 01:40:42.266661 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.266503 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" event={"ID":"ddf96e2d-efcd-4be9-86f2-4219f32ebb39","Type":"ContainerDied","Data":"b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e"} Sep 16 01:40:42.266661 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.266544 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9" event={"ID":"ddf96e2d-efcd-4be9-86f2-4219f32ebb39","Type":"ContainerDied","Data":"2b7e10ac9d061bef97ea7336e44b693dee67e9434869444ced550eba1a0eb2f8"} Sep 16 01:40:42.266661 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.266561 2567 scope.go:117] "RemoveContainer" containerID="b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e" Sep 16 01:40:42.275846 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.275828 2567 scope.go:117] "RemoveContainer" containerID="b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e" Sep 16 01:40:42.276088 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:40:42.276067 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e\": container with ID starting with b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e not found: ID does not exist" containerID="b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e" Sep 16 01:40:42.276157 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.276095 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e"} err="failed to get container status \"b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e\": rpc error: code = NotFound desc = could not find container \"b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e\": container with ID starting with b0421c5c21297c9640b0c46d96714ac22bcf9540440bc032cc5a0b6c7f98726e not found: ID does not exist" Sep 16 01:40:42.284165 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.284140 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9"] Sep 16 01:40:42.287243 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:42.287221 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/auth-test-model-cbb499-kserve-59b8f965f5-lq8c9"] Sep 16 01:40:43.939501 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:43.939468 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddf96e2d-efcd-4be9-86f2-4219f32ebb39" path="/var/lib/kubelet/pods/ddf96e2d-efcd-4be9-86f2-4219f32ebb39/volumes" Sep 16 01:40:44.328136 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:44.328098 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:44.328346 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:44.328154 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:44.340898 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:44.340870 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:45.294345 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:45.294308 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:40:53.524099 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.524062 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn"] Sep 16 01:40:53.524637 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.524293 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" podUID="9b28645e-88c8-4adc-8269-4258dc3fe7e0" containerName="maas-api" containerID="cri-o://9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260" gracePeriod=30 Sep 16 01:40:53.772787 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.772754 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:40:53.837587 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.837500 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-maas-api-tls\") pod \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " Sep 16 01:40:53.837587 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.837556 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-metrics-certs\") pod \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " Sep 16 01:40:53.837766 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.837660 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q6km\" (UniqueName: \"kubernetes.io/projected/9b28645e-88c8-4adc-8269-4258dc3fe7e0-kube-api-access-8q6km\") pod \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " Sep 16 01:40:53.837766 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.837760 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b28645e-88c8-4adc-8269-4258dc3fe7e0-openshift-service-ca\") pod \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\" (UID: \"9b28645e-88c8-4adc-8269-4258dc3fe7e0\") " Sep 16 01:40:53.838178 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.838130 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b28645e-88c8-4adc-8269-4258dc3fe7e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b28645e-88c8-4adc-8269-4258dc3fe7e0" (UID: "9b28645e-88c8-4adc-8269-4258dc3fe7e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:40:53.839841 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.839814 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b28645e-88c8-4adc-8269-4258dc3fe7e0" (UID: "9b28645e-88c8-4adc-8269-4258dc3fe7e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:40:53.840380 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.840347 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9b28645e-88c8-4adc-8269-4258dc3fe7e0" (UID: "9b28645e-88c8-4adc-8269-4258dc3fe7e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:40:53.840471 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.840373 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b28645e-88c8-4adc-8269-4258dc3fe7e0-kube-api-access-8q6km" (OuterVolumeSpecName: "kube-api-access-8q6km") pod "9b28645e-88c8-4adc-8269-4258dc3fe7e0" (UID: "9b28645e-88c8-4adc-8269-4258dc3fe7e0"). InnerVolumeSpecName "kube-api-access-8q6km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:40:53.939317 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.939259 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q6km\" (UniqueName: \"kubernetes.io/projected/9b28645e-88c8-4adc-8269-4258dc3fe7e0-kube-api-access-8q6km\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:40:53.939317 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.939314 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b28645e-88c8-4adc-8269-4258dc3fe7e0-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:40:53.939518 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.939329 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:40:53.939518 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:53.939342 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b28645e-88c8-4adc-8269-4258dc3fe7e0-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:40:54.332047 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:54.332012 2567 generic.go:356] "Generic (PLEG): container finished" podID="9b28645e-88c8-4adc-8269-4258dc3fe7e0" containerID="9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260" exitCode=0 Sep 16 01:40:54.332238 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:54.332068 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" Sep 16 01:40:54.332238 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:54.332108 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" event={"ID":"9b28645e-88c8-4adc-8269-4258dc3fe7e0","Type":"ContainerDied","Data":"9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260"} Sep 16 01:40:54.332238 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:54.332176 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn" event={"ID":"9b28645e-88c8-4adc-8269-4258dc3fe7e0","Type":"ContainerDied","Data":"12e64963a2cfa3ed6f6a0c0fe95988ef856ac8e7197c46e02ed8aec5a5fcc857"} Sep 16 01:40:54.332238 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:54.332205 2567 scope.go:117] "RemoveContainer" containerID="9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260" Sep 16 01:40:54.342468 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:54.342449 2567 scope.go:117] "RemoveContainer" containerID="9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260" Sep 16 01:40:54.342746 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:40:54.342723 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260\": container with ID starting with 9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260 not found: ID does not exist" containerID="9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260" Sep 16 01:40:54.342809 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:54.342758 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260"} err="failed to get container status \"9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260\": rpc error: code = NotFound desc = could not find container \"9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260\": container with ID starting with 9078f7b797bd936d51969bbd423e8521499b77246c6bf598cb5cf93c6efe4260 not found: ID does not exist" Sep 16 01:40:54.348908 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:54.348884 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn"] Sep 16 01:40:54.352136 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:54.352116 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3b6e6489-557f5d7b5b-vzvzn"] Sep 16 01:40:55.940065 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:40:55.940027 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b28645e-88c8-4adc-8269-4258dc3fe7e0" path="/var/lib/kubelet/pods/9b28645e-88c8-4adc-8269-4258dc3fe7e0/volumes" Sep 16 01:41:31.745265 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:31.745226 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7"] Sep 16 01:41:36.613889 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.613844 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz"] Sep 16 01:41:36.614512 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.614491 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b28645e-88c8-4adc-8269-4258dc3fe7e0" containerName="maas-api" Sep 16 01:41:36.614598 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.614515 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b28645e-88c8-4adc-8269-4258dc3fe7e0" containerName="maas-api" Sep 16 01:41:36.614598 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.614529 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddf96e2d-efcd-4be9-86f2-4219f32ebb39" containerName="main" Sep 16 01:41:36.614598 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.614537 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddf96e2d-efcd-4be9-86f2-4219f32ebb39" containerName="main" Sep 16 01:41:36.614757 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.614641 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddf96e2d-efcd-4be9-86f2-4219f32ebb39" containerName="main" Sep 16 01:41:36.614757 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.614654 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b28645e-88c8-4adc-8269-4258dc3fe7e0" containerName="maas-api" Sep 16 01:41:36.617904 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.617871 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.620655 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.620635 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-cbb499\"/\"rate-test-model-cbb499-kserve-self-signed-certs\"" Sep 16 01:41:36.624491 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.624468 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz"] Sep 16 01:41:36.780048 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.780010 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfsp\" (UniqueName: \"kubernetes.io/projected/3650bb20-284d-4520-a4b6-59aa4f46d8e3-kube-api-access-zpfsp\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.780234 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.780088 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-model-cache\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.780234 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.780116 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tmp-dir\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.780234 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.780131 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-dshm\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.780234 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.780160 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tls-certs\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.780234 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.780211 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-home\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.881078 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.880998 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfsp\" (UniqueName: \"kubernetes.io/projected/3650bb20-284d-4520-a4b6-59aa4f46d8e3-kube-api-access-zpfsp\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.881078 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.881067 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-model-cache\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.881260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.881093 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tmp-dir\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.881260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.881109 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-dshm\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.881260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.881141 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tls-certs\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.881260 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.881165 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-home\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.881656 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.881623 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-model-cache\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.881788 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.881662 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-home\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.881788 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.881674 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tmp-dir\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.883603 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.883583 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-dshm\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.883848 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.883829 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tls-certs\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.890400 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.890379 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfsp\" (UniqueName: \"kubernetes.io/projected/3650bb20-284d-4520-a4b6-59aa4f46d8e3-kube-api-access-zpfsp\") pod \"rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:36.928420 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:36.928377 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:37.200684 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:37.200655 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz"] Sep 16 01:41:37.202113 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:41:37.202087 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3650bb20_284d_4520_a4b6_59aa4f46d8e3.slice/crio-1b9d72380cde842d33e4154008bccf4929b96cc9fc12f4956655d2b2f7ccff58 WatchSource:0}: Error finding container 1b9d72380cde842d33e4154008bccf4929b96cc9fc12f4956655d2b2f7ccff58: Status 404 returned error can't find the container with id 1b9d72380cde842d33e4154008bccf4929b96cc9fc12f4956655d2b2f7ccff58 Sep 16 01:41:37.546618 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:37.546580 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" event={"ID":"3650bb20-284d-4520-a4b6-59aa4f46d8e3","Type":"ContainerStarted","Data":"19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442"} Sep 16 01:41:37.546829 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:37.546632 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" event={"ID":"3650bb20-284d-4520-a4b6-59aa4f46d8e3","Type":"ContainerStarted","Data":"1b9d72380cde842d33e4154008bccf4929b96cc9fc12f4956655d2b2f7ccff58"} Sep 16 01:41:37.564597 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:37.564542 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" podStartSLOduration=1.564524024 podStartE2EDuration="1.564524024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:41:37.562593342 +0000 UTC m=+1548.148395552" watchObservedRunningTime="2026-09-16 01:41:37.564524024 +0000 UTC m=+1548.150326235" Sep 16 01:41:46.765294 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:46.765235 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" podUID="7d3453bb-6596-456e-87cb-c742cbe85d53" containerName="main" containerID="cri-o://43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202" gracePeriod=45 Sep 16 01:41:46.928617 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:46.928585 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:46.928748 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:46.928658 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:46.941548 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:46.941517 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:47.015948 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.015881 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:41:47.085797 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.085761 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcgdv\" (UniqueName: \"kubernetes.io/projected/7d3453bb-6596-456e-87cb-c742cbe85d53-kube-api-access-xcgdv\") pod \"7d3453bb-6596-456e-87cb-c742cbe85d53\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " Sep 16 01:41:47.086019 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.085844 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-model-cache\") pod \"7d3453bb-6596-456e-87cb-c742cbe85d53\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " Sep 16 01:41:47.086019 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.085924 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-dshm\") pod \"7d3453bb-6596-456e-87cb-c742cbe85d53\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " Sep 16 01:41:47.086019 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.085949 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-tmp-dir\") pod \"7d3453bb-6596-456e-87cb-c742cbe85d53\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " Sep 16 01:41:47.086019 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.085972 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-home\") pod \"7d3453bb-6596-456e-87cb-c742cbe85d53\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " Sep 16 01:41:47.086019 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.085990 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3453bb-6596-456e-87cb-c742cbe85d53-tls-certs\") pod \"7d3453bb-6596-456e-87cb-c742cbe85d53\" (UID: \"7d3453bb-6596-456e-87cb-c742cbe85d53\") " Sep 16 01:41:47.086354 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.086163 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d3453bb-6596-456e-87cb-c742cbe85d53" (UID: "7d3453bb-6596-456e-87cb-c742cbe85d53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:47.086354 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.086298 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d3453bb-6596-456e-87cb-c742cbe85d53" (UID: "7d3453bb-6596-456e-87cb-c742cbe85d53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:47.086354 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.086322 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-home" (OuterVolumeSpecName: "home") pod "7d3453bb-6596-456e-87cb-c742cbe85d53" (UID: "7d3453bb-6596-456e-87cb-c742cbe85d53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:47.086514 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.086368 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-model-cache\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:41:47.088194 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.088168 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-dshm" (OuterVolumeSpecName: "dshm") pod "7d3453bb-6596-456e-87cb-c742cbe85d53" (UID: "7d3453bb-6596-456e-87cb-c742cbe85d53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:41:47.088312 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.088171 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3453bb-6596-456e-87cb-c742cbe85d53-kube-api-access-xcgdv" (OuterVolumeSpecName: "kube-api-access-xcgdv") pod "7d3453bb-6596-456e-87cb-c742cbe85d53" (UID: "7d3453bb-6596-456e-87cb-c742cbe85d53"). InnerVolumeSpecName "kube-api-access-xcgdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:41:47.088312 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.088203 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d3453bb-6596-456e-87cb-c742cbe85d53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d3453bb-6596-456e-87cb-c742cbe85d53" (UID: "7d3453bb-6596-456e-87cb-c742cbe85d53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:41:47.187092 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.187047 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d3453bb-6596-456e-87cb-c742cbe85d53-tls-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:41:47.187092 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.187090 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcgdv\" (UniqueName: \"kubernetes.io/projected/7d3453bb-6596-456e-87cb-c742cbe85d53-kube-api-access-xcgdv\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:41:47.187092 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.187100 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-dshm\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:41:47.187092 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.187110 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-tmp-dir\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:41:47.187398 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.187118 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d3453bb-6596-456e-87cb-c742cbe85d53-home\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:41:47.601011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.600975 2567 generic.go:356] "Generic (PLEG): container finished" podID="7d3453bb-6596-456e-87cb-c742cbe85d53" containerID="43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202" exitCode=0 Sep 16 01:41:47.601189 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.601023 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" Sep 16 01:41:47.601189 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.601068 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" event={"ID":"7d3453bb-6596-456e-87cb-c742cbe85d53","Type":"ContainerDied","Data":"43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202"} Sep 16 01:41:47.601189 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.601114 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7" event={"ID":"7d3453bb-6596-456e-87cb-c742cbe85d53","Type":"ContainerDied","Data":"69b56bbd083e0fc258205068c919208de7374fd844bcc5d3c25c53f290776f91"} Sep 16 01:41:47.601189 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.601134 2567 scope.go:117] "RemoveContainer" containerID="43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202" Sep 16 01:41:47.610473 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.610455 2567 scope.go:117] "RemoveContainer" containerID="43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202" Sep 16 01:41:47.610768 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:41:47.610748 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202\": container with ID starting with 43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202 not found: ID does not exist" containerID="43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202" Sep 16 01:41:47.610839 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.610777 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202"} err="failed to get container status \"43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202\": rpc error: code = NotFound desc = could not find container \"43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202\": container with ID starting with 43fe00efda909f0d77c422030afb54f7a7edd6fadacc8c2fb5eda4417fef9202 not found: ID does not exist" Sep 16 01:41:47.613338 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.613318 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:41:47.622255 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.622231 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7"] Sep 16 01:41:47.624069 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.624047 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/sub-test-model-cbb499-kserve-7dd57f5c65-87wv7"] Sep 16 01:41:47.939558 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:41:47.939469 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d3453bb-6596-456e-87cb-c742cbe85d53" path="/var/lib/kubelet/pods/7d3453bb-6596-456e-87cb-c742cbe85d53/volumes" Sep 16 01:42:40.894679 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:40.894593 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz"] Sep 16 01:42:48.041073 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.041036 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp"] Sep 16 01:42:48.041748 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.041716 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d3453bb-6596-456e-87cb-c742cbe85d53" containerName="main" Sep 16 01:42:48.041748 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.041739 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d3453bb-6596-456e-87cb-c742cbe85d53" containerName="main" Sep 16 01:42:48.041895 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.041860 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d3453bb-6596-456e-87cb-c742cbe85d53" containerName="main" Sep 16 01:42:48.045149 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.045131 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.047378 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.047353 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8d230b-serving-cert\"" Sep 16 01:42:48.058146 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.058118 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp"] Sep 16 01:42:48.180849 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.180811 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-metrics-certs\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.181018 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.180857 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n42tz\" (UniqueName: \"kubernetes.io/projected/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-kube-api-access-n42tz\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.181018 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.180967 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.181018 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.181015 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.282467 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.282425 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-metrics-certs\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.282661 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.282474 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n42tz\" (UniqueName: \"kubernetes.io/projected/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-kube-api-access-n42tz\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.282661 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.282561 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.282661 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.282624 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.283240 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.283215 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.285033 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.285010 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.285119 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.285094 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-metrics-certs\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.289920 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.289900 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n42tz\" (UniqueName: \"kubernetes.io/projected/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-kube-api-access-n42tz\") pod \"maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.359180 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.359099 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:48.636252 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.636224 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp"] Sep 16 01:42:48.638739 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:42:48.638708 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182f2bdf_91f8_458f_9c7b_a127c0f7cba1.slice/crio-77dded926f7d510f094435be916160e29483fbf6e830fca59054a3b2eebdb5e6 WatchSource:0}: Error finding container 77dded926f7d510f094435be916160e29483fbf6e830fca59054a3b2eebdb5e6: Status 404 returned error can't find the container with id 77dded926f7d510f094435be916160e29483fbf6e830fca59054a3b2eebdb5e6 Sep 16 01:42:48.904876 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:48.904789 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" event={"ID":"182f2bdf-91f8-458f-9c7b-a127c0f7cba1","Type":"ContainerStarted","Data":"77dded926f7d510f094435be916160e29483fbf6e830fca59054a3b2eebdb5e6"} Sep 16 01:42:49.910913 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:49.910880 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" event={"ID":"182f2bdf-91f8-458f-9c7b-a127c0f7cba1","Type":"ContainerStarted","Data":"4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f"} Sep 16 01:42:49.911337 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:49.910984 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:49.928564 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:49.928511 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" podStartSLOduration=1.507701984 podStartE2EDuration="1.928494824s" totalImagesPullingTime="420.79284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:42:49.927177516 +0000 UTC m=+1620.512979729" watchObservedRunningTime="2026-09-16 01:42:49.928494824 +0000 UTC m=+1620.514297034" Sep 16 01:42:50.653060 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.653021 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w"] Sep 16 01:42:50.657228 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.657202 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:50.659792 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.659766 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7d0959-serving-cert\"" Sep 16 01:42:50.668201 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.668163 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w"] Sep 16 01:42:50.808297 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.808231 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdx48\" (UniqueName: \"kubernetes.io/projected/42c2e818-755c-4513-b493-ecee7763af58-kube-api-access-zdx48\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:50.808477 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.808306 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-metrics-certs\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:50.808477 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.808336 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:50.808477 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.808379 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c2e818-755c-4513-b493-ecee7763af58-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:50.909985 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.909905 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdx48\" (UniqueName: \"kubernetes.io/projected/42c2e818-755c-4513-b493-ecee7763af58-kube-api-access-zdx48\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:50.909985 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.909950 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-metrics-certs\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:50.910186 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.909987 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:50.910186 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.910032 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c2e818-755c-4513-b493-ecee7763af58-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:50.910186 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:42:50.910132 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-serving-cert: secret "maas-api-e2e-api-a-7d0959-serving-cert" not found Sep 16 01:42:50.910416 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:42:50.910219 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls podName:42c2e818-755c-4513-b493-ecee7763af58 nodeName:}" failed. No retries permitted until 2026-09-16 01:42:51.410198264 +0000 UTC m=+1621.996000455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls") pod "maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" (UID: "42c2e818-755c-4513-b493-ecee7763af58") : secret "maas-api-e2e-api-a-7d0959-serving-cert" not found Sep 16 01:42:50.910817 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.910795 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c2e818-755c-4513-b493-ecee7763af58-openshift-service-ca\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:50.912812 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.912785 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-metrics-certs\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:50.918739 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:50.918712 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdx48\" (UniqueName: \"kubernetes.io/projected/42c2e818-755c-4513-b493-ecee7763af58-kube-api-access-zdx48\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:51.415029 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:51.414985 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:51.415216 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:42:51.415163 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-serving-cert: secret "maas-api-e2e-api-a-7d0959-serving-cert" not found Sep 16 01:42:51.415268 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:42:51.415244 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls podName:42c2e818-755c-4513-b493-ecee7763af58 nodeName:}" failed. No retries permitted until 2026-09-16 01:42:52.415223268 +0000 UTC m=+1623.001025471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls") pod "maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" (UID: "42c2e818-755c-4513-b493-ecee7763af58") : secret "maas-api-e2e-api-a-7d0959-serving-cert" not found Sep 16 01:42:52.432755 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:52.432717 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:52.435152 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:52.435131 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls\") pod \"maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:52.471574 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:52.471537 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:52.743837 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:52.743806 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w"] Sep 16 01:42:52.745557 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:42:52.745525 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c2e818_755c_4513_b493_ecee7763af58.slice/crio-f3e8d75cce6a0aec73bfb9cd948f31c3bc33f44a89f71dcfc37f3df670636d23 WatchSource:0}: Error finding container f3e8d75cce6a0aec73bfb9cd948f31c3bc33f44a89f71dcfc37f3df670636d23: Status 404 returned error can't find the container with id f3e8d75cce6a0aec73bfb9cd948f31c3bc33f44a89f71dcfc37f3df670636d23 Sep 16 01:42:52.927306 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:52.927242 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" event={"ID":"42c2e818-755c-4513-b493-ecee7763af58","Type":"ContainerStarted","Data":"f3e8d75cce6a0aec73bfb9cd948f31c3bc33f44a89f71dcfc37f3df670636d23"} Sep 16 01:42:53.938720 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:53.938692 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:42:53.939087 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:53.938739 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" event={"ID":"42c2e818-755c-4513-b493-ecee7763af58","Type":"ContainerStarted","Data":"248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b"} Sep 16 01:42:53.951573 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:53.951528 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" podStartSLOduration=3.6050106299999998 podStartE2EDuration="3.9515129s" totalImagesPullingTime="346.50227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:42:53.949715568 +0000 UTC m=+1624.535517780" watchObservedRunningTime="2026-09-16 01:42:53.9515129 +0000 UTC m=+1624.537315111" Sep 16 01:42:55.914662 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:55.914623 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" podUID="3650bb20-284d-4520-a4b6-59aa4f46d8e3" containerName="main" containerID="cri-o://19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442" gracePeriod=45 Sep 16 01:42:55.921480 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:55.921457 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:42:56.166384 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.166327 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:42:56.272034 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.271994 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpfsp\" (UniqueName: \"kubernetes.io/projected/3650bb20-284d-4520-a4b6-59aa4f46d8e3-kube-api-access-zpfsp\") pod \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " Sep 16 01:42:56.272209 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.272047 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tmp-dir\") pod \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " Sep 16 01:42:56.272209 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.272064 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tls-certs\") pod \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " Sep 16 01:42:56.272209 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.272088 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-dshm\") pod \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " Sep 16 01:42:56.272386 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.272207 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-model-cache\") pod \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " Sep 16 01:42:56.272386 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.272256 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-home\") pod \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\" (UID: \"3650bb20-284d-4520-a4b6-59aa4f46d8e3\") " Sep 16 01:42:56.272386 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.272347 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3650bb20-284d-4520-a4b6-59aa4f46d8e3" (UID: "3650bb20-284d-4520-a4b6-59aa4f46d8e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:42:56.272553 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.272515 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3650bb20-284d-4520-a4b6-59aa4f46d8e3" (UID: "3650bb20-284d-4520-a4b6-59aa4f46d8e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:42:56.272665 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.272632 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-home" (OuterVolumeSpecName: "home") pod "3650bb20-284d-4520-a4b6-59aa4f46d8e3" (UID: "3650bb20-284d-4520-a4b6-59aa4f46d8e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:42:56.272796 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.272759 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tmp-dir\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:42:56.272796 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.272774 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-model-cache\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:42:56.272796 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.272790 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-home\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:42:56.274503 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.274473 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3650bb20-284d-4520-a4b6-59aa4f46d8e3-kube-api-access-zpfsp" (OuterVolumeSpecName: "kube-api-access-zpfsp") pod "3650bb20-284d-4520-a4b6-59aa4f46d8e3" (UID: "3650bb20-284d-4520-a4b6-59aa4f46d8e3"). InnerVolumeSpecName "kube-api-access-zpfsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:42:56.274579 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.274505 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3650bb20-284d-4520-a4b6-59aa4f46d8e3" (UID: "3650bb20-284d-4520-a4b6-59aa4f46d8e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:42:56.274579 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.274505 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-dshm" (OuterVolumeSpecName: "dshm") pod "3650bb20-284d-4520-a4b6-59aa4f46d8e3" (UID: "3650bb20-284d-4520-a4b6-59aa4f46d8e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 01:42:56.374173 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.374140 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpfsp\" (UniqueName: \"kubernetes.io/projected/3650bb20-284d-4520-a4b6-59aa4f46d8e3-kube-api-access-zpfsp\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:42:56.374173 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.374169 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3650bb20-284d-4520-a4b6-59aa4f46d8e3-tls-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:42:56.374173 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.374180 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3650bb20-284d-4520-a4b6-59aa4f46d8e3-dshm\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:42:56.948893 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.948860 2567 generic.go:356] "Generic (PLEG): container finished" podID="3650bb20-284d-4520-a4b6-59aa4f46d8e3" containerID="19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442" exitCode=0 Sep 16 01:42:56.949349 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.948903 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" Sep 16 01:42:56.949349 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.948948 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" event={"ID":"3650bb20-284d-4520-a4b6-59aa4f46d8e3","Type":"ContainerDied","Data":"19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442"} Sep 16 01:42:56.949349 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.948990 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz" event={"ID":"3650bb20-284d-4520-a4b6-59aa4f46d8e3","Type":"ContainerDied","Data":"1b9d72380cde842d33e4154008bccf4929b96cc9fc12f4956655d2b2f7ccff58"} Sep 16 01:42:56.949349 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.949007 2567 scope.go:117] "RemoveContainer" containerID="19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442" Sep 16 01:42:56.958507 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.958490 2567 scope.go:117] "RemoveContainer" containerID="19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442" Sep 16 01:42:56.958789 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:42:56.958771 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442\": container with ID starting with 19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442 not found: ID does not exist" containerID="19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442" Sep 16 01:42:56.958839 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.958800 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442"} err="failed to get container status \"19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442\": rpc error: code = NotFound desc = could not find container \"19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442\": container with ID starting with 19556fba0f5b3a1cc87776d3e81904ae665d3bc86ffb1fafa7b994f9147b3442 not found: ID does not exist" Sep 16 01:42:56.969360 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.969323 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz"] Sep 16 01:42:56.972503 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:56.972472 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cbb499/rate-test-model-cbb499-kserve-7696cb9c5f-vlqdz"] Sep 16 01:42:57.939971 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:57.939934 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3650bb20-284d-4520-a4b6-59aa4f46d8e3" path="/var/lib/kubelet/pods/3650bb20-284d-4520-a4b6-59aa4f46d8e3/volumes" Sep 16 01:42:59.944750 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:42:59.944725 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:43:18.142538 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.142500 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl"] Sep 16 01:43:18.142940 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.142928 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3650bb20-284d-4520-a4b6-59aa4f46d8e3" containerName="main" Sep 16 01:43:18.142991 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.142942 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3650bb20-284d-4520-a4b6-59aa4f46d8e3" containerName="main" Sep 16 01:43:18.143033 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.143010 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3650bb20-284d-4520-a4b6-59aa4f46d8e3" containerName="main" Sep 16 01:43:18.146617 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.146596 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.148972 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.148942 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a42101-serving-cert\"" Sep 16 01:43:18.157364 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.157299 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl"] Sep 16 01:43:18.293878 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.293843 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzl77\" (UniqueName: \"kubernetes.io/projected/4ddb2492-0829-4518-8fac-6205a79a0d36-kube-api-access-mzl77\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.293878 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.293884 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.294106 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.293969 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ddb2492-0829-4518-8fac-6205a79a0d36-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.294106 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.294061 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-metrics-certs\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.394802 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.394703 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzl77\" (UniqueName: \"kubernetes.io/projected/4ddb2492-0829-4518-8fac-6205a79a0d36-kube-api-access-mzl77\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.394802 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.394759 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.395036 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.394826 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ddb2492-0829-4518-8fac-6205a79a0d36-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.395036 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.394884 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-metrics-certs\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.395817 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.395684 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ddb2492-0829-4518-8fac-6205a79a0d36-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.397828 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.397803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-metrics-certs\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.397965 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.397803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.409601 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.409566 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzl77\" (UniqueName: \"kubernetes.io/projected/4ddb2492-0829-4518-8fac-6205a79a0d36-kube-api-access-mzl77\") pod \"maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.463653 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.463618 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:18.743149 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:18.743114 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl"] Sep 16 01:43:18.745864 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:43:18.745832 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddb2492_0829_4518_8fac_6205a79a0d36.slice/crio-45dd6557d29c0b0d3e39ac8b39b0af734670bd2d8af1e4c82a4603415e1d1c0e WatchSource:0}: Error finding container 45dd6557d29c0b0d3e39ac8b39b0af734670bd2d8af1e4c82a4603415e1d1c0e: Status 404 returned error can't find the container with id 45dd6557d29c0b0d3e39ac8b39b0af734670bd2d8af1e4c82a4603415e1d1c0e Sep 16 01:43:19.060387 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:19.060300 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" event={"ID":"4ddb2492-0829-4518-8fac-6205a79a0d36","Type":"ContainerStarted","Data":"45dd6557d29c0b0d3e39ac8b39b0af734670bd2d8af1e4c82a4603415e1d1c0e"} Sep 16 01:43:20.066630 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:20.066591 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" event={"ID":"4ddb2492-0829-4518-8fac-6205a79a0d36","Type":"ContainerStarted","Data":"dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b"} Sep 16 01:43:20.067107 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:20.066652 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:20.084561 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:20.084507 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" podStartSLOduration=1.7004039789999998 podStartE2EDuration="2.084491882s" totalImagesPullingTime="384.087903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:43:20.081894182 +0000 UTC m=+1650.667696397" watchObservedRunningTime="2026-09-16 01:43:20.084491882 +0000 UTC m=+1650.670294092" Sep 16 01:43:26.077700 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:26.077667 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:43:27.330331 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.330269 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd"] Sep 16 01:43:27.334367 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.334334 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.336644 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.336617 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4e1413-serving-cert\"" Sep 16 01:43:27.346130 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.346104 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd"] Sep 16 01:43:27.381796 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.381758 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-maas-api-tls\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.381796 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.381794 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djdm5\" (UniqueName: \"kubernetes.io/projected/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-kube-api-access-djdm5\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.382030 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.381926 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.382030 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.381988 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-metrics-certs\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.482475 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.482437 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.482696 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.482499 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-metrics-certs\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.482696 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.482540 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-maas-api-tls\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.482696 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.482559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdm5\" (UniqueName: \"kubernetes.io/projected/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-kube-api-access-djdm5\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.482696 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:43:27.482677 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-serving-cert: secret "maas-api-e2e-api-b-4e1413-serving-cert" not found Sep 16 01:43:27.482910 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:43:27.482769 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-maas-api-tls podName:6540cdd4-aaeb-410c-a000-6cdafe7fb80b nodeName:}" failed. No retries permitted until 2026-09-16 01:43:27.982748082 +0000 UTC m=+1658.568550286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-maas-api-tls") pod "maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" (UID: "6540cdd4-aaeb-410c-a000-6cdafe7fb80b") : secret "maas-api-e2e-api-b-4e1413-serving-cert" not found Sep 16 01:43:27.483200 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.483176 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.485157 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.485137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-metrics-certs\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.490697 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.490677 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdm5\" (UniqueName: \"kubernetes.io/projected/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-kube-api-access-djdm5\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.988411 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.988376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-maas-api-tls\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:27.991033 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:27.991006 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-maas-api-tls\") pod \"maas-api-e2e-api-b-4e1413-5994b747b5-psfgd\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:28.249614 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:28.249523 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:28.521087 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:28.521059 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd"] Sep 16 01:43:28.522782 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:43:28.522757 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6540cdd4_aaeb_410c_a000_6cdafe7fb80b.slice/crio-ec9679f3f2a7187828f7669a1fbaa85bc916448b1ce170efa6e32a36b3bae533 WatchSource:0}: Error finding container ec9679f3f2a7187828f7669a1fbaa85bc916448b1ce170efa6e32a36b3bae533: Status 404 returned error can't find the container with id ec9679f3f2a7187828f7669a1fbaa85bc916448b1ce170efa6e32a36b3bae533 Sep 16 01:43:29.115474 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:29.115376 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" event={"ID":"6540cdd4-aaeb-410c-a000-6cdafe7fb80b","Type":"ContainerStarted","Data":"0c4241d1e4e22548a904632722d42e1bea8932b64d68df031a7b0165f17d91f7"} Sep 16 01:43:29.115474 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:29.115439 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" event={"ID":"6540cdd4-aaeb-410c-a000-6cdafe7fb80b","Type":"ContainerStarted","Data":"ec9679f3f2a7187828f7669a1fbaa85bc916448b1ce170efa6e32a36b3bae533"} Sep 16 01:43:29.115702 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:29.115519 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:29.132931 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:29.132881 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" podStartSLOduration=1.8199404860000001 podStartE2EDuration="2.132867918s" totalImagesPullingTime="312.927432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:43:29.131921379 +0000 UTC m=+1659.717723588" watchObservedRunningTime="2026-09-16 01:43:29.132867918 +0000 UTC m=+1659.718670129" Sep 16 01:43:35.127121 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:35.127091 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:43:50.120590 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:43:50.120552 2567 scope.go:117] "RemoveContainer" containerID="d2478ebad6d1cd2aaddd13ff42646cb4cc3206752009299eb1df4630a1ff46b4" Sep 16 01:44:10.784786 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:10.784692 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w"] Sep 16 01:44:10.785405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:10.784969 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" podUID="42c2e818-755c-4513-b493-ecee7763af58" containerName="maas-api" containerID="cri-o://248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b" gracePeriod=30 Sep 16 01:44:11.036483 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.036423 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:44:11.120532 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.120492 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdx48\" (UniqueName: \"kubernetes.io/projected/42c2e818-755c-4513-b493-ecee7763af58-kube-api-access-zdx48\") pod \"42c2e818-755c-4513-b493-ecee7763af58\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " Sep 16 01:44:11.120702 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.120539 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c2e818-755c-4513-b493-ecee7763af58-openshift-service-ca\") pod \"42c2e818-755c-4513-b493-ecee7763af58\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " Sep 16 01:44:11.120702 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.120657 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-metrics-certs\") pod \"42c2e818-755c-4513-b493-ecee7763af58\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " Sep 16 01:44:11.120793 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.120759 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls\") pod \"42c2e818-755c-4513-b493-ecee7763af58\" (UID: \"42c2e818-755c-4513-b493-ecee7763af58\") " Sep 16 01:44:11.120883 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.120854 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c2e818-755c-4513-b493-ecee7763af58-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42c2e818-755c-4513-b493-ecee7763af58" (UID: "42c2e818-755c-4513-b493-ecee7763af58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:44:11.121124 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.121098 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c2e818-755c-4513-b493-ecee7763af58-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:11.122867 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.122846 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "42c2e818-755c-4513-b493-ecee7763af58" (UID: "42c2e818-755c-4513-b493-ecee7763af58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:44:11.122995 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.122933 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42c2e818-755c-4513-b493-ecee7763af58" (UID: "42c2e818-755c-4513-b493-ecee7763af58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:44:11.122995 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.122981 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c2e818-755c-4513-b493-ecee7763af58-kube-api-access-zdx48" (OuterVolumeSpecName: "kube-api-access-zdx48") pod "42c2e818-755c-4513-b493-ecee7763af58" (UID: "42c2e818-755c-4513-b493-ecee7763af58"). InnerVolumeSpecName "kube-api-access-zdx48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:44:11.222495 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.222456 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:11.222495 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.222489 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c2e818-755c-4513-b493-ecee7763af58-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:11.222495 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.222499 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdx48\" (UniqueName: \"kubernetes.io/projected/42c2e818-755c-4513-b493-ecee7763af58-kube-api-access-zdx48\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:11.347382 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.347271 2567 generic.go:356] "Generic (PLEG): container finished" podID="42c2e818-755c-4513-b493-ecee7763af58" containerID="248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b" exitCode=0 Sep 16 01:44:11.347382 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.347333 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" Sep 16 01:44:11.347605 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.347338 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" event={"ID":"42c2e818-755c-4513-b493-ecee7763af58","Type":"ContainerDied","Data":"248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b"} Sep 16 01:44:11.347605 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.347465 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w" event={"ID":"42c2e818-755c-4513-b493-ecee7763af58","Type":"ContainerDied","Data":"f3e8d75cce6a0aec73bfb9cd948f31c3bc33f44a89f71dcfc37f3df670636d23"} Sep 16 01:44:11.347605 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.347496 2567 scope.go:117] "RemoveContainer" containerID="248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b" Sep 16 01:44:11.357064 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.357047 2567 scope.go:117] "RemoveContainer" containerID="248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b" Sep 16 01:44:11.357353 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:44:11.357324 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b\": container with ID starting with 248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b not found: ID does not exist" containerID="248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b" Sep 16 01:44:11.357451 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.357364 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b"} err="failed to get container status \"248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b\": rpc error: code = NotFound desc = could not find container \"248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b\": container with ID starting with 248a499578d0ac530ea7d7d10048b7400c5ce5ad7996c11a814af5e258026c6b not found: ID does not exist" Sep 16 01:44:11.378006 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.377980 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w"] Sep 16 01:44:11.382057 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.382029 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7d0959-76cdfd7fb4-td88w"] Sep 16 01:44:11.939899 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:11.939862 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c2e818-755c-4513-b493-ecee7763af58" path="/var/lib/kubelet/pods/42c2e818-755c-4513-b493-ecee7763af58/volumes" Sep 16 01:44:30.323906 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.323863 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd"] Sep 16 01:44:30.324412 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.324122 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" podUID="6540cdd4-aaeb-410c-a000-6cdafe7fb80b" containerName="maas-api" containerID="cri-o://0c4241d1e4e22548a904632722d42e1bea8932b64d68df031a7b0165f17d91f7" gracePeriod=30 Sep 16 01:44:30.459870 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.459757 2567 generic.go:356] "Generic (PLEG): container finished" podID="6540cdd4-aaeb-410c-a000-6cdafe7fb80b" containerID="0c4241d1e4e22548a904632722d42e1bea8932b64d68df031a7b0165f17d91f7" exitCode=0 Sep 16 01:44:30.459870 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.459806 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" event={"ID":"6540cdd4-aaeb-410c-a000-6cdafe7fb80b","Type":"ContainerDied","Data":"0c4241d1e4e22548a904632722d42e1bea8932b64d68df031a7b0165f17d91f7"} Sep 16 01:44:30.577644 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.577586 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:44:30.721792 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.721762 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-maas-api-tls\") pod \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " Sep 16 01:44:30.721995 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.721806 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djdm5\" (UniqueName: \"kubernetes.io/projected/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-kube-api-access-djdm5\") pod \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " Sep 16 01:44:30.721995 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.721850 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-metrics-certs\") pod \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " Sep 16 01:44:30.721995 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.721961 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-openshift-service-ca\") pod \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\" (UID: \"6540cdd4-aaeb-410c-a000-6cdafe7fb80b\") " Sep 16 01:44:30.722433 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.722404 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6540cdd4-aaeb-410c-a000-6cdafe7fb80b" (UID: "6540cdd4-aaeb-410c-a000-6cdafe7fb80b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:44:30.724152 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.724124 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6540cdd4-aaeb-410c-a000-6cdafe7fb80b" (UID: "6540cdd4-aaeb-410c-a000-6cdafe7fb80b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:44:30.724310 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.724128 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-kube-api-access-djdm5" (OuterVolumeSpecName: "kube-api-access-djdm5") pod "6540cdd4-aaeb-410c-a000-6cdafe7fb80b" (UID: "6540cdd4-aaeb-410c-a000-6cdafe7fb80b"). InnerVolumeSpecName "kube-api-access-djdm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:44:30.724517 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.724496 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6540cdd4-aaeb-410c-a000-6cdafe7fb80b" (UID: "6540cdd4-aaeb-410c-a000-6cdafe7fb80b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:44:30.823418 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.823382 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:30.823418 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.823416 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:30.823623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.823429 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djdm5\" (UniqueName: \"kubernetes.io/projected/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-kube-api-access-djdm5\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:30.823623 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:30.823439 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6540cdd4-aaeb-410c-a000-6cdafe7fb80b-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:31.466694 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:31.466657 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" Sep 16 01:44:31.466694 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:31.466686 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd" event={"ID":"6540cdd4-aaeb-410c-a000-6cdafe7fb80b","Type":"ContainerDied","Data":"ec9679f3f2a7187828f7669a1fbaa85bc916448b1ce170efa6e32a36b3bae533"} Sep 16 01:44:31.467173 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:31.466754 2567 scope.go:117] "RemoveContainer" containerID="0c4241d1e4e22548a904632722d42e1bea8932b64d68df031a7b0165f17d91f7" Sep 16 01:44:31.488761 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:31.488728 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd"] Sep 16 01:44:31.490535 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:31.490507 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4e1413-5994b747b5-psfgd"] Sep 16 01:44:31.940292 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:31.940248 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6540cdd4-aaeb-410c-a000-6cdafe7fb80b" path="/var/lib/kubelet/pods/6540cdd4-aaeb-410c-a000-6cdafe7fb80b/volumes" Sep 16 01:44:34.450347 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.450307 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh"] Sep 16 01:44:34.450754 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.450740 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6540cdd4-aaeb-410c-a000-6cdafe7fb80b" containerName="maas-api" Sep 16 01:44:34.450804 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.450756 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6540cdd4-aaeb-410c-a000-6cdafe7fb80b" containerName="maas-api" Sep 16 01:44:34.450804 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.450770 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c2e818-755c-4513-b493-ecee7763af58" containerName="maas-api" Sep 16 01:44:34.450804 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.450775 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c2e818-755c-4513-b493-ecee7763af58" containerName="maas-api" Sep 16 01:44:34.450892 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.450846 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c2e818-755c-4513-b493-ecee7763af58" containerName="maas-api" Sep 16 01:44:34.450892 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.450855 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6540cdd4-aaeb-410c-a000-6cdafe7fb80b" containerName="maas-api" Sep 16 01:44:34.454607 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.454582 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:34.456197 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.456175 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6pzk\" (UniqueName: \"kubernetes.io/projected/1e48ac20-22c8-44c1-903c-905a54d97b0e-kube-api-access-w6pzk\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:34.456399 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.456234 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:34.456399 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.456311 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:34.456399 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.456334 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e48ac20-22c8-44c1-903c-905a54d97b0e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:34.457344 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.457327 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-f73a36-serving-cert\"" Sep 16 01:44:34.464337 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.464312 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh"] Sep 16 01:44:34.557570 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.557525 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:34.557769 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.557604 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:34.557769 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.557632 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e48ac20-22c8-44c1-903c-905a54d97b0e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:34.557769 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.557731 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6pzk\" (UniqueName: \"kubernetes.io/projected/1e48ac20-22c8-44c1-903c-905a54d97b0e-kube-api-access-w6pzk\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:34.557769 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:44:34.557752 2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-serving-cert: secret "maas-api-e2e-ipp-cleanup-f73a36-serving-cert" not found Sep 16 01:44:34.557993 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:44:34.557828 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-maas-api-tls podName:1e48ac20-22c8-44c1-903c-905a54d97b0e nodeName:}" failed. No retries permitted until 2026-09-16 01:44:35.057805388 +0000 UTC m=+1725.643607577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" (UID: "1e48ac20-22c8-44c1-903c-905a54d97b0e") : secret "maas-api-e2e-ipp-cleanup-f73a36-serving-cert" not found Sep 16 01:44:34.558362 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.558345 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e48ac20-22c8-44c1-903c-905a54d97b0e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:34.560120 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.560099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:34.570345 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:34.570322 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6pzk\" (UniqueName: \"kubernetes.io/projected/1e48ac20-22c8-44c1-903c-905a54d97b0e-kube-api-access-w6pzk\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:35.062397 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:35.062359 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:35.064920 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:35.064890 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:35.069860 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:35.069833 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:35.346018 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:35.345992 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh"] Sep 16 01:44:35.348155 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:44:35.348133 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e48ac20_22c8_44c1_903c_905a54d97b0e.slice/crio-67f2f5ceba5d2cfc270f65dfd6ec229ac27011cdc8be52a59b932debf805b2e5 WatchSource:0}: Error finding container 67f2f5ceba5d2cfc270f65dfd6ec229ac27011cdc8be52a59b932debf805b2e5: Status 404 returned error can't find the container with id 67f2f5ceba5d2cfc270f65dfd6ec229ac27011cdc8be52a59b932debf805b2e5 Sep 16 01:44:35.349552 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:35.349534 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 01:44:35.491148 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:35.491113 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" event={"ID":"1e48ac20-22c8-44c1-903c-905a54d97b0e","Type":"ContainerStarted","Data":"67f2f5ceba5d2cfc270f65dfd6ec229ac27011cdc8be52a59b932debf805b2e5"} Sep 16 01:44:36.497402 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:36.497360 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" event={"ID":"1e48ac20-22c8-44c1-903c-905a54d97b0e","Type":"ContainerStarted","Data":"3e3a0788c3fb70a938d8360a3c3308d97a4aa9a8f2172a7c8f2c1f8f71ccceab"} Sep 16 01:44:36.497835 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:36.497499 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:36.515636 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:36.515578 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" podStartSLOduration=2.167503111 podStartE2EDuration="2.515556523s" totalImagesPullingTime="348.053412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:44:36.512778356 +0000 UTC m=+1727.098580568" watchObservedRunningTime="2026-09-16 01:44:36.515556523 +0000 UTC m=+1727.101358769" Sep 16 01:44:42.508375 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:42.508335 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:44:48.772928 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:48.772892 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf"] Sep 16 01:44:48.773428 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:48.773128 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" podUID="b29099ae-0d49-4ec1-9441-2aec696cc971" containerName="maas-api" containerID="cri-o://5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d" gracePeriod=30 Sep 16 01:44:49.026387 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.026325 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:44:49.112817 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.112781 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-maas-api-tls\") pod \"b29099ae-0d49-4ec1-9441-2aec696cc971\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " Sep 16 01:44:49.112817 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.112815 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b29099ae-0d49-4ec1-9441-2aec696cc971-openshift-service-ca\") pod \"b29099ae-0d49-4ec1-9441-2aec696cc971\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " Sep 16 01:44:49.113080 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.112850 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kckf\" (UniqueName: \"kubernetes.io/projected/b29099ae-0d49-4ec1-9441-2aec696cc971-kube-api-access-6kckf\") pod \"b29099ae-0d49-4ec1-9441-2aec696cc971\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " Sep 16 01:44:49.113080 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.112970 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-metrics-certs\") pod \"b29099ae-0d49-4ec1-9441-2aec696cc971\" (UID: \"b29099ae-0d49-4ec1-9441-2aec696cc971\") " Sep 16 01:44:49.113232 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.113205 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29099ae-0d49-4ec1-9441-2aec696cc971-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b29099ae-0d49-4ec1-9441-2aec696cc971" (UID: "b29099ae-0d49-4ec1-9441-2aec696cc971"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:44:49.113379 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.113362 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b29099ae-0d49-4ec1-9441-2aec696cc971-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:49.115159 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.115128 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b29099ae-0d49-4ec1-9441-2aec696cc971" (UID: "b29099ae-0d49-4ec1-9441-2aec696cc971"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:44:49.115593 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.115573 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b29099ae-0d49-4ec1-9441-2aec696cc971" (UID: "b29099ae-0d49-4ec1-9441-2aec696cc971"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:44:49.115593 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.115579 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29099ae-0d49-4ec1-9441-2aec696cc971-kube-api-access-6kckf" (OuterVolumeSpecName: "kube-api-access-6kckf") pod "b29099ae-0d49-4ec1-9441-2aec696cc971" (UID: "b29099ae-0d49-4ec1-9441-2aec696cc971"). InnerVolumeSpecName "kube-api-access-6kckf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:44:49.214480 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.214447 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:49.214480 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.214475 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b29099ae-0d49-4ec1-9441-2aec696cc971-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:49.214480 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.214486 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kckf\" (UniqueName: \"kubernetes.io/projected/b29099ae-0d49-4ec1-9441-2aec696cc971-kube-api-access-6kckf\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:44:49.568693 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.568655 2567 generic.go:356] "Generic (PLEG): container finished" podID="b29099ae-0d49-4ec1-9441-2aec696cc971" containerID="5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d" exitCode=0 Sep 16 01:44:49.568896 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.568715 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" Sep 16 01:44:49.568896 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.568736 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" event={"ID":"b29099ae-0d49-4ec1-9441-2aec696cc971","Type":"ContainerDied","Data":"5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d"} Sep 16 01:44:49.568896 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.568795 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf" event={"ID":"b29099ae-0d49-4ec1-9441-2aec696cc971","Type":"ContainerDied","Data":"54c63609614ecb4c617fbb1b1b4c0d8689d60f6f8eba9f16c26445bc01992e3d"} Sep 16 01:44:49.568896 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.568825 2567 scope.go:117] "RemoveContainer" containerID="5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d" Sep 16 01:44:49.579096 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.579079 2567 scope.go:117] "RemoveContainer" containerID="5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d" Sep 16 01:44:49.579429 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:44:49.579403 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d\": container with ID starting with 5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d not found: ID does not exist" containerID="5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d" Sep 16 01:44:49.579490 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.579443 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d"} err="failed to get container status \"5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d\": rpc error: code = NotFound desc = could not find container \"5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d\": container with ID starting with 5d662d255047abab2ccf5c345dce24ac44f79e752365f0e325149650768bb72d not found: ID does not exist" Sep 16 01:44:49.589964 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.589942 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf"] Sep 16 01:44:49.593798 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.593777 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-4bc0fb-675d9cbc45-c92kf"] Sep 16 01:44:49.940715 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:44:49.940632 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b29099ae-0d49-4ec1-9441-2aec696cc971" path="/var/lib/kubelet/pods/b29099ae-0d49-4ec1-9441-2aec696cc971/volumes" Sep 16 01:45:02.445317 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.445261 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh"] Sep 16 01:45:02.445795 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.445519 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" podUID="1e48ac20-22c8-44c1-903c-905a54d97b0e" containerName="maas-api" containerID="cri-o://3e3a0788c3fb70a938d8360a3c3308d97a4aa9a8f2172a7c8f2c1f8f71ccceab" gracePeriod=30 Sep 16 01:45:02.503240 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.503192 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" podUID="1e48ac20-22c8-44c1-903c-905a54d97b0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.66:8443/health\": dial tcp 10.132.0.66:8443: connect: connection refused" Sep 16 01:45:02.639373 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.639335 2567 generic.go:356] "Generic (PLEG): container finished" podID="1e48ac20-22c8-44c1-903c-905a54d97b0e" containerID="3e3a0788c3fb70a938d8360a3c3308d97a4aa9a8f2172a7c8f2c1f8f71ccceab" exitCode=0 Sep 16 01:45:02.639545 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.639453 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" event={"ID":"1e48ac20-22c8-44c1-903c-905a54d97b0e","Type":"ContainerDied","Data":"3e3a0788c3fb70a938d8360a3c3308d97a4aa9a8f2172a7c8f2c1f8f71ccceab"} Sep 16 01:45:02.708492 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.708424 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:45:02.853356 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.853326 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-metrics-certs\") pod \"1e48ac20-22c8-44c1-903c-905a54d97b0e\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " Sep 16 01:45:02.853547 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.853394 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-maas-api-tls\") pod \"1e48ac20-22c8-44c1-903c-905a54d97b0e\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " Sep 16 01:45:02.853547 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.853451 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e48ac20-22c8-44c1-903c-905a54d97b0e-openshift-service-ca\") pod \"1e48ac20-22c8-44c1-903c-905a54d97b0e\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " Sep 16 01:45:02.853547 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.853477 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6pzk\" (UniqueName: \"kubernetes.io/projected/1e48ac20-22c8-44c1-903c-905a54d97b0e-kube-api-access-w6pzk\") pod \"1e48ac20-22c8-44c1-903c-905a54d97b0e\" (UID: \"1e48ac20-22c8-44c1-903c-905a54d97b0e\") " Sep 16 01:45:02.853836 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.853806 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e48ac20-22c8-44c1-903c-905a54d97b0e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e48ac20-22c8-44c1-903c-905a54d97b0e" (UID: "1e48ac20-22c8-44c1-903c-905a54d97b0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:45:02.855739 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.855707 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1e48ac20-22c8-44c1-903c-905a54d97b0e" (UID: "1e48ac20-22c8-44c1-903c-905a54d97b0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:45:02.855851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.855783 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e48ac20-22c8-44c1-903c-905a54d97b0e-kube-api-access-w6pzk" (OuterVolumeSpecName: "kube-api-access-w6pzk") pod "1e48ac20-22c8-44c1-903c-905a54d97b0e" (UID: "1e48ac20-22c8-44c1-903c-905a54d97b0e"). InnerVolumeSpecName "kube-api-access-w6pzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:45:02.855851 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.855798 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e48ac20-22c8-44c1-903c-905a54d97b0e" (UID: "1e48ac20-22c8-44c1-903c-905a54d97b0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:45:02.955314 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.955264 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:02.955314 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.955309 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e48ac20-22c8-44c1-903c-905a54d97b0e-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:02.955314 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.955318 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e48ac20-22c8-44c1-903c-905a54d97b0e-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:02.955544 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:02.955328 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6pzk\" (UniqueName: \"kubernetes.io/projected/1e48ac20-22c8-44c1-903c-905a54d97b0e-kube-api-access-w6pzk\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:03.645405 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:03.645368 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" event={"ID":"1e48ac20-22c8-44c1-903c-905a54d97b0e","Type":"ContainerDied","Data":"67f2f5ceba5d2cfc270f65dfd6ec229ac27011cdc8be52a59b932debf805b2e5"} Sep 16 01:45:03.645870 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:03.645425 2567 scope.go:117] "RemoveContainer" containerID="3e3a0788c3fb70a938d8360a3c3308d97a4aa9a8f2172a7c8f2c1f8f71ccceab" Sep 16 01:45:03.645870 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:03.645382 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh" Sep 16 01:45:03.670141 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:03.670106 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh"] Sep 16 01:45:03.673566 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:03.673541 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f73a36-7cfc6d686d-7drkh"] Sep 16 01:45:03.939716 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:03.939635 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e48ac20-22c8-44c1-903c-905a54d97b0e" path="/var/lib/kubelet/pods/1e48ac20-22c8-44c1-903c-905a54d97b0e/volumes" Sep 16 01:45:06.775104 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:06.775072 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s"] Sep 16 01:45:06.775583 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:06.775387 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" podUID="1c32416f-e4bd-4be7-9378-5561418569c9" containerName="maas-api" containerID="cri-o://adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f" gracePeriod=30 Sep 16 01:45:07.025496 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.025439 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:45:07.097175 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.097142 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c32416f-e4bd-4be7-9378-5561418569c9-openshift-service-ca\") pod \"1c32416f-e4bd-4be7-9378-5561418569c9\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " Sep 16 01:45:07.097376 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.097195 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-maas-api-tls\") pod \"1c32416f-e4bd-4be7-9378-5561418569c9\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " Sep 16 01:45:07.097376 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.097228 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-metrics-certs\") pod \"1c32416f-e4bd-4be7-9378-5561418569c9\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " Sep 16 01:45:07.097510 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.097379 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29jcm\" (UniqueName: \"kubernetes.io/projected/1c32416f-e4bd-4be7-9378-5561418569c9-kube-api-access-29jcm\") pod \"1c32416f-e4bd-4be7-9378-5561418569c9\" (UID: \"1c32416f-e4bd-4be7-9378-5561418569c9\") " Sep 16 01:45:07.097633 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.097611 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c32416f-e4bd-4be7-9378-5561418569c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1c32416f-e4bd-4be7-9378-5561418569c9" (UID: "1c32416f-e4bd-4be7-9378-5561418569c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:45:07.097786 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.097768 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c32416f-e4bd-4be7-9378-5561418569c9-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:07.099606 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.099572 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c32416f-e4bd-4be7-9378-5561418569c9-kube-api-access-29jcm" (OuterVolumeSpecName: "kube-api-access-29jcm") pod "1c32416f-e4bd-4be7-9378-5561418569c9" (UID: "1c32416f-e4bd-4be7-9378-5561418569c9"). InnerVolumeSpecName "kube-api-access-29jcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:45:07.099606 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.099583 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1c32416f-e4bd-4be7-9378-5561418569c9" (UID: "1c32416f-e4bd-4be7-9378-5561418569c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:45:07.099758 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.099744 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1c32416f-e4bd-4be7-9378-5561418569c9" (UID: "1c32416f-e4bd-4be7-9378-5561418569c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:45:07.198670 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.198618 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29jcm\" (UniqueName: \"kubernetes.io/projected/1c32416f-e4bd-4be7-9378-5561418569c9-kube-api-access-29jcm\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:07.198670 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.198655 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:07.198670 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.198665 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c32416f-e4bd-4be7-9378-5561418569c9-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:07.671450 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.671417 2567 generic.go:356] "Generic (PLEG): container finished" podID="1c32416f-e4bd-4be7-9378-5561418569c9" containerID="adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f" exitCode=0 Sep 16 01:45:07.671629 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.671466 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" Sep 16 01:45:07.671629 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.671533 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" event={"ID":"1c32416f-e4bd-4be7-9378-5561418569c9","Type":"ContainerDied","Data":"adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f"} Sep 16 01:45:07.671629 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.671593 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s" event={"ID":"1c32416f-e4bd-4be7-9378-5561418569c9","Type":"ContainerDied","Data":"59fc70db24cdf4325a162c5947782742014bb8a9be41ceeb4813bb2bb8454352"} Sep 16 01:45:07.671629 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.671627 2567 scope.go:117] "RemoveContainer" containerID="adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f" Sep 16 01:45:07.682236 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.682217 2567 scope.go:117] "RemoveContainer" containerID="adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f" Sep 16 01:45:07.682543 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:45:07.682524 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f\": container with ID starting with adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f not found: ID does not exist" containerID="adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f" Sep 16 01:45:07.682604 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.682550 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f"} err="failed to get container status \"adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f\": rpc error: code = NotFound desc = could not find container \"adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f\": container with ID starting with adfc239bca3e57417f275bf6834c61403afe5f7113dab9defaef9ee156d11e6f not found: ID does not exist" Sep 16 01:45:07.692769 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.692747 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s"] Sep 16 01:45:07.696459 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.696440 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-0d2655-6694547b79-mv89s"] Sep 16 01:45:07.940017 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:07.939938 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c32416f-e4bd-4be7-9378-5561418569c9" path="/var/lib/kubelet/pods/1c32416f-e4bd-4be7-9378-5561418569c9/volumes" Sep 16 01:45:28.796929 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:28.796891 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp"] Sep 16 01:45:28.797466 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:28.797162 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" podUID="182f2bdf-91f8-458f-9c7b-a127c0f7cba1" containerName="maas-api" containerID="cri-o://4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f" gracePeriod=30 Sep 16 01:45:29.042243 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.042218 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:45:29.216507 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.216424 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-metrics-certs\") pod \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " Sep 16 01:45:29.216507 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.216486 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-maas-api-tls\") pod \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " Sep 16 01:45:29.216723 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.216555 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n42tz\" (UniqueName: \"kubernetes.io/projected/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-kube-api-access-n42tz\") pod \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " Sep 16 01:45:29.216723 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.216610 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-openshift-service-ca\") pod \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\" (UID: \"182f2bdf-91f8-458f-9c7b-a127c0f7cba1\") " Sep 16 01:45:29.217048 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.217020 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "182f2bdf-91f8-458f-9c7b-a127c0f7cba1" (UID: "182f2bdf-91f8-458f-9c7b-a127c0f7cba1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:45:29.218836 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.218813 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-kube-api-access-n42tz" (OuterVolumeSpecName: "kube-api-access-n42tz") pod "182f2bdf-91f8-458f-9c7b-a127c0f7cba1" (UID: "182f2bdf-91f8-458f-9c7b-a127c0f7cba1"). InnerVolumeSpecName "kube-api-access-n42tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:45:29.219153 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.219134 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "182f2bdf-91f8-458f-9c7b-a127c0f7cba1" (UID: "182f2bdf-91f8-458f-9c7b-a127c0f7cba1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:45:29.219212 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.219159 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "182f2bdf-91f8-458f-9c7b-a127c0f7cba1" (UID: "182f2bdf-91f8-458f-9c7b-a127c0f7cba1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:45:29.317630 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.317601 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:29.317630 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.317628 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n42tz\" (UniqueName: \"kubernetes.io/projected/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-kube-api-access-n42tz\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:29.317815 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.317638 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:29.317815 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.317647 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182f2bdf-91f8-458f-9c7b-a127c0f7cba1-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:45:29.782472 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.782435 2567 generic.go:356] "Generic (PLEG): container finished" podID="182f2bdf-91f8-458f-9c7b-a127c0f7cba1" containerID="4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f" exitCode=0 Sep 16 01:45:29.782660 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.782515 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" Sep 16 01:45:29.782660 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.782537 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" event={"ID":"182f2bdf-91f8-458f-9c7b-a127c0f7cba1","Type":"ContainerDied","Data":"4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f"} Sep 16 01:45:29.782660 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.782619 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp" event={"ID":"182f2bdf-91f8-458f-9c7b-a127c0f7cba1","Type":"ContainerDied","Data":"77dded926f7d510f094435be916160e29483fbf6e830fca59054a3b2eebdb5e6"} Sep 16 01:45:29.782660 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.782643 2567 scope.go:117] "RemoveContainer" containerID="4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f" Sep 16 01:45:29.792831 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.792810 2567 scope.go:117] "RemoveContainer" containerID="4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f" Sep 16 01:45:29.793154 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:45:29.793122 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f\": container with ID starting with 4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f not found: ID does not exist" containerID="4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f" Sep 16 01:45:29.793252 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.793165 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f"} err="failed to get container status \"4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f\": rpc error: code = NotFound desc = could not find container \"4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f\": container with ID starting with 4dc7e439a7a8fa3f727dece2ad1fa645b85600af20c347eb4790f9bdb340c80f not found: ID does not exist" Sep 16 01:45:29.808020 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.807993 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp"] Sep 16 01:45:29.810152 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.810126 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8d230b-7bb95b94fd-kv5tp"] Sep 16 01:45:29.939583 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:45:29.939547 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="182f2bdf-91f8-458f-9c7b-a127c0f7cba1" path="/var/lib/kubelet/pods/182f2bdf-91f8-458f-9c7b-a127c0f7cba1/volumes" Sep 16 01:46:28.039012 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.038964 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl"] Sep 16 01:46:28.039629 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.039224 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" podUID="4ddb2492-0829-4518-8fac-6205a79a0d36" containerName="maas-api" containerID="cri-o://dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b" gracePeriod=30 Sep 16 01:46:28.295988 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.295929 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:46:28.375014 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.374988 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-maas-api-tls\") pod \"4ddb2492-0829-4518-8fac-6205a79a0d36\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " Sep 16 01:46:28.375201 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.375025 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzl77\" (UniqueName: \"kubernetes.io/projected/4ddb2492-0829-4518-8fac-6205a79a0d36-kube-api-access-mzl77\") pod \"4ddb2492-0829-4518-8fac-6205a79a0d36\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " Sep 16 01:46:28.375201 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.375162 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ddb2492-0829-4518-8fac-6205a79a0d36-openshift-service-ca\") pod \"4ddb2492-0829-4518-8fac-6205a79a0d36\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " Sep 16 01:46:28.375343 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.375238 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-metrics-certs\") pod \"4ddb2492-0829-4518-8fac-6205a79a0d36\" (UID: \"4ddb2492-0829-4518-8fac-6205a79a0d36\") " Sep 16 01:46:28.375553 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.375512 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ddb2492-0829-4518-8fac-6205a79a0d36-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ddb2492-0829-4518-8fac-6205a79a0d36" (UID: "4ddb2492-0829-4518-8fac-6205a79a0d36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:46:28.375698 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.375621 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ddb2492-0829-4518-8fac-6205a79a0d36-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:46:28.377504 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.377482 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ddb2492-0829-4518-8fac-6205a79a0d36" (UID: "4ddb2492-0829-4518-8fac-6205a79a0d36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:46:28.377504 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.377482 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddb2492-0829-4518-8fac-6205a79a0d36-kube-api-access-mzl77" (OuterVolumeSpecName: "kube-api-access-mzl77") pod "4ddb2492-0829-4518-8fac-6205a79a0d36" (UID: "4ddb2492-0829-4518-8fac-6205a79a0d36"). InnerVolumeSpecName "kube-api-access-mzl77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:46:28.377631 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.377548 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4ddb2492-0829-4518-8fac-6205a79a0d36" (UID: "4ddb2492-0829-4518-8fac-6205a79a0d36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:46:28.476267 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.476234 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:46:28.476267 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.476264 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ddb2492-0829-4518-8fac-6205a79a0d36-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:46:28.476496 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:28.476299 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzl77\" (UniqueName: \"kubernetes.io/projected/4ddb2492-0829-4518-8fac-6205a79a0d36-kube-api-access-mzl77\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:46:29.072705 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:29.072669 2567 generic.go:356] "Generic (PLEG): container finished" podID="4ddb2492-0829-4518-8fac-6205a79a0d36" containerID="dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b" exitCode=0 Sep 16 01:46:29.073114 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:29.072733 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" Sep 16 01:46:29.073114 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:29.072722 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" event={"ID":"4ddb2492-0829-4518-8fac-6205a79a0d36","Type":"ContainerDied","Data":"dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b"} Sep 16 01:46:29.073114 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:29.072811 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl" event={"ID":"4ddb2492-0829-4518-8fac-6205a79a0d36","Type":"ContainerDied","Data":"45dd6557d29c0b0d3e39ac8b39b0af734670bd2d8af1e4c82a4603415e1d1c0e"} Sep 16 01:46:29.073114 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:29.072842 2567 scope.go:117] "RemoveContainer" containerID="dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b" Sep 16 01:46:29.082238 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:29.082216 2567 scope.go:117] "RemoveContainer" containerID="dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b" Sep 16 01:46:29.082564 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:46:29.082542 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b\": container with ID starting with dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b not found: ID does not exist" containerID="dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b" Sep 16 01:46:29.082635 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:29.082572 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b"} err="failed to get container status \"dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b\": rpc error: code = NotFound desc = could not find container \"dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b\": container with ID starting with dd32bf97306306430fae8f9ae2464872d2505af1494aee79fedfb26e39e2741b not found: ID does not exist" Sep 16 01:46:29.094895 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:29.094864 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl"] Sep 16 01:46:29.096699 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:29.096679 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a42101-7f78fc577c-txmgl"] Sep 16 01:46:29.938915 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:29.938881 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ddb2492-0829-4518-8fac-6205a79a0d36" path="/var/lib/kubelet/pods/4ddb2492-0829-4518-8fac-6205a79a0d36/volumes" Sep 16 01:46:45.310980 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.310942 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws"] Sep 16 01:46:45.311432 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.311185 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" podUID="fa1c0515-cf74-4231-9141-8028cc2a601f" containerName="maas-api" containerID="cri-o://6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32" gracePeriod=30 Sep 16 01:46:45.567484 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.567422 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:46:45.645369 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.645330 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1c0515-cf74-4231-9141-8028cc2a601f-openshift-service-ca\") pod \"fa1c0515-cf74-4231-9141-8028cc2a601f\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " Sep 16 01:46:45.645563 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.645376 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-metrics-certs\") pod \"fa1c0515-cf74-4231-9141-8028cc2a601f\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " Sep 16 01:46:45.645563 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.645450 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfwnj\" (UniqueName: \"kubernetes.io/projected/fa1c0515-cf74-4231-9141-8028cc2a601f-kube-api-access-cfwnj\") pod \"fa1c0515-cf74-4231-9141-8028cc2a601f\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " Sep 16 01:46:45.645563 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.645481 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-maas-api-tls\") pod \"fa1c0515-cf74-4231-9141-8028cc2a601f\" (UID: \"fa1c0515-cf74-4231-9141-8028cc2a601f\") " Sep 16 01:46:45.645715 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.645677 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1c0515-cf74-4231-9141-8028cc2a601f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fa1c0515-cf74-4231-9141-8028cc2a601f" (UID: "fa1c0515-cf74-4231-9141-8028cc2a601f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:46:45.645810 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.645793 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa1c0515-cf74-4231-9141-8028cc2a601f-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:46:45.647671 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.647643 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fa1c0515-cf74-4231-9141-8028cc2a601f" (UID: "fa1c0515-cf74-4231-9141-8028cc2a601f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:46:45.647817 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.647793 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1c0515-cf74-4231-9141-8028cc2a601f-kube-api-access-cfwnj" (OuterVolumeSpecName: "kube-api-access-cfwnj") pod "fa1c0515-cf74-4231-9141-8028cc2a601f" (UID: "fa1c0515-cf74-4231-9141-8028cc2a601f"). InnerVolumeSpecName "kube-api-access-cfwnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:46:45.647817 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.647805 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fa1c0515-cf74-4231-9141-8028cc2a601f" (UID: "fa1c0515-cf74-4231-9141-8028cc2a601f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:46:45.746539 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.746502 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:46:45.746539 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.746534 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwnj\" (UniqueName: \"kubernetes.io/projected/fa1c0515-cf74-4231-9141-8028cc2a601f-kube-api-access-cfwnj\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:46:45.746539 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:45.746544 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa1c0515-cf74-4231-9141-8028cc2a601f-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:46:46.157069 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:46.157034 2567 generic.go:356] "Generic (PLEG): container finished" podID="fa1c0515-cf74-4231-9141-8028cc2a601f" containerID="6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32" exitCode=0 Sep 16 01:46:46.157235 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:46.157082 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" Sep 16 01:46:46.157235 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:46.157109 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" event={"ID":"fa1c0515-cf74-4231-9141-8028cc2a601f","Type":"ContainerDied","Data":"6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32"} Sep 16 01:46:46.157235 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:46.157151 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws" event={"ID":"fa1c0515-cf74-4231-9141-8028cc2a601f","Type":"ContainerDied","Data":"02636e308c0aae43d63bf27e98ad2df6a74f5dd6179fb57d1e890ded6b504fcc"} Sep 16 01:46:46.157235 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:46.157171 2567 scope.go:117] "RemoveContainer" containerID="6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32" Sep 16 01:46:46.166205 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:46.166185 2567 scope.go:117] "RemoveContainer" containerID="6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32" Sep 16 01:46:46.166533 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:46:46.166511 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32\": container with ID starting with 6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32 not found: ID does not exist" containerID="6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32" Sep 16 01:46:46.166613 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:46.166542 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32"} err="failed to get container status \"6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32\": rpc error: code = NotFound desc = could not find container \"6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32\": container with ID starting with 6a757fc236fbd99671c386115ed4ebac2183f7aca1e0f5c6654da6f78c90cd32 not found: ID does not exist" Sep 16 01:46:46.173974 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:46.173944 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws"] Sep 16 01:46:46.175781 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:46.175757 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cbb499-64b59ccd8d-mwbws"] Sep 16 01:46:47.939666 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:46:47.939631 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa1c0515-cf74-4231-9141-8028cc2a601f" path="/var/lib/kubelet/pods/fa1c0515-cf74-4231-9141-8028cc2a601f/volumes" Sep 16 01:47:03.046256 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.046215 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7"] Sep 16 01:47:03.046880 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.046487 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" podUID="e6390385-8fb5-4c70-b967-fc444957e86f" containerName="maas-api" containerID="cri-o://619a150b91884dcdacf292980af83c80eea348a2e99cc461d3dff30e55830a85" gracePeriod=30 Sep 16 01:47:03.245892 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.245860 2567 generic.go:356] "Generic (PLEG): container finished" podID="e6390385-8fb5-4c70-b967-fc444957e86f" containerID="619a150b91884dcdacf292980af83c80eea348a2e99cc461d3dff30e55830a85" exitCode=0 Sep 16 01:47:03.246040 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.245910 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" event={"ID":"e6390385-8fb5-4c70-b967-fc444957e86f","Type":"ContainerDied","Data":"619a150b91884dcdacf292980af83c80eea348a2e99cc461d3dff30e55830a85"} Sep 16 01:47:03.292973 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.292936 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:47:03.308907 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.308836 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6390385-8fb5-4c70-b967-fc444957e86f-openshift-service-ca\") pod \"e6390385-8fb5-4c70-b967-fc444957e86f\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " Sep 16 01:47:03.308907 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.308884 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7xfk\" (UniqueName: \"kubernetes.io/projected/e6390385-8fb5-4c70-b967-fc444957e86f-kube-api-access-m7xfk\") pod \"e6390385-8fb5-4c70-b967-fc444957e86f\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " Sep 16 01:47:03.309118 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.308985 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-metrics-certs\") pod \"e6390385-8fb5-4c70-b967-fc444957e86f\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " Sep 16 01:47:03.309118 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.309010 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-maas-api-tls\") pod \"e6390385-8fb5-4c70-b967-fc444957e86f\" (UID: \"e6390385-8fb5-4c70-b967-fc444957e86f\") " Sep 16 01:47:03.309323 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.309243 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6390385-8fb5-4c70-b967-fc444957e86f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6390385-8fb5-4c70-b967-fc444957e86f" (UID: "e6390385-8fb5-4c70-b967-fc444957e86f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 01:47:03.311253 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.311226 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e6390385-8fb5-4c70-b967-fc444957e86f" (UID: "e6390385-8fb5-4c70-b967-fc444957e86f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:47:03.311414 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.311256 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6390385-8fb5-4c70-b967-fc444957e86f-kube-api-access-m7xfk" (OuterVolumeSpecName: "kube-api-access-m7xfk") pod "e6390385-8fb5-4c70-b967-fc444957e86f" (UID: "e6390385-8fb5-4c70-b967-fc444957e86f"). InnerVolumeSpecName "kube-api-access-m7xfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:47:03.311414 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.311272 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6390385-8fb5-4c70-b967-fc444957e86f" (UID: "e6390385-8fb5-4c70-b967-fc444957e86f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:47:03.410439 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.410406 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:47:03.410439 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.410434 2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6390385-8fb5-4c70-b967-fc444957e86f-maas-api-tls\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:47:03.410439 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.410445 2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6390385-8fb5-4c70-b967-fc444957e86f-openshift-service-ca\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:47:03.410751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:03.410455 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7xfk\" (UniqueName: \"kubernetes.io/projected/e6390385-8fb5-4c70-b967-fc444957e86f-kube-api-access-m7xfk\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:47:04.251458 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:04.251428 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" Sep 16 01:47:04.251838 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:04.251456 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7" event={"ID":"e6390385-8fb5-4c70-b967-fc444957e86f","Type":"ContainerDied","Data":"a7f95847e13d3f3a2538cea92a0443b6e22d627ecb5f0b90786116cf1e3b3462"} Sep 16 01:47:04.251838 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:04.251517 2567 scope.go:117] "RemoveContainer" containerID="619a150b91884dcdacf292980af83c80eea348a2e99cc461d3dff30e55830a85" Sep 16 01:47:04.267366 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:04.267341 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7"] Sep 16 01:47:04.271264 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:04.271239 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c0ec73-6ccb6b9795-fg4h7"] Sep 16 01:47:05.939047 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:05.939011 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6390385-8fb5-4c70-b967-fc444957e86f" path="/var/lib/kubelet/pods/e6390385-8fb5-4c70-b967-fc444957e86f/volumes" Sep 16 01:47:12.062001 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.061892 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6859d7476b-n4bnr"] Sep 16 01:47:12.289264 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.289223 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6859d7476b-n4bnr" podUID="36927d41-75a7-4d07-b192-bdaf5294c32f" containerName="manager" containerID="cri-o://aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542" gracePeriod=30 Sep 16 01:47:12.539671 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.539642 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:47:12.594067 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.594036 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-metrics-certs\") pod \"36927d41-75a7-4d07-b192-bdaf5294c32f\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " Sep 16 01:47:12.594067 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.594073 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lz27\" (UniqueName: \"kubernetes.io/projected/36927d41-75a7-4d07-b192-bdaf5294c32f-kube-api-access-6lz27\") pod \"36927d41-75a7-4d07-b192-bdaf5294c32f\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " Sep 16 01:47:12.594310 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.594100 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-webhook-cert\") pod \"36927d41-75a7-4d07-b192-bdaf5294c32f\" (UID: \"36927d41-75a7-4d07-b192-bdaf5294c32f\") " Sep 16 01:47:12.596267 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.596236 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "36927d41-75a7-4d07-b192-bdaf5294c32f" (UID: "36927d41-75a7-4d07-b192-bdaf5294c32f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:47:12.596809 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.596786 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "36927d41-75a7-4d07-b192-bdaf5294c32f" (UID: "36927d41-75a7-4d07-b192-bdaf5294c32f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:47:12.596809 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.596798 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36927d41-75a7-4d07-b192-bdaf5294c32f-kube-api-access-6lz27" (OuterVolumeSpecName: "kube-api-access-6lz27") pod "36927d41-75a7-4d07-b192-bdaf5294c32f" (UID: "36927d41-75a7-4d07-b192-bdaf5294c32f"). InnerVolumeSpecName "kube-api-access-6lz27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:47:12.695011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.694935 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:47:12.695011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.694962 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lz27\" (UniqueName: \"kubernetes.io/projected/36927d41-75a7-4d07-b192-bdaf5294c32f-kube-api-access-6lz27\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:47:12.695011 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:12.694971 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36927d41-75a7-4d07-b192-bdaf5294c32f-webhook-cert\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:47:13.295517 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:13.295476 2567 generic.go:356] "Generic (PLEG): container finished" podID="36927d41-75a7-4d07-b192-bdaf5294c32f" containerID="aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542" exitCode=0 Sep 16 01:47:13.295928 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:13.295535 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-n4bnr" Sep 16 01:47:13.295928 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:13.295566 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-n4bnr" event={"ID":"36927d41-75a7-4d07-b192-bdaf5294c32f","Type":"ContainerDied","Data":"aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542"} Sep 16 01:47:13.295928 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:13.295625 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-n4bnr" event={"ID":"36927d41-75a7-4d07-b192-bdaf5294c32f","Type":"ContainerDied","Data":"b99d2246dfc0d0c8ee1ffec663ec357f4f6921f8225491461f84d29222eda19c"} Sep 16 01:47:13.295928 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:13.295649 2567 scope.go:117] "RemoveContainer" containerID="aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542" Sep 16 01:47:13.305996 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:13.305964 2567 scope.go:117] "RemoveContainer" containerID="aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542" Sep 16 01:47:13.306315 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:47:13.306296 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542\": container with ID starting with aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542 not found: ID does not exist" containerID="aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542" Sep 16 01:47:13.306384 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:13.306325 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542"} err="failed to get container status \"aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542\": rpc error: code = NotFound desc = could not find container \"aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542\": container with ID starting with aba0e8b792434d66c39731606fa2d397cbeb24b9f4787620c2c59af384f57542 not found: ID does not exist" Sep 16 01:47:13.317250 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:13.317224 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6859d7476b-n4bnr"] Sep 16 01:47:13.323186 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:13.323165 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6859d7476b-n4bnr"] Sep 16 01:47:13.939367 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:47:13.939323 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36927d41-75a7-4d07-b192-bdaf5294c32f" path="/var/lib/kubelet/pods/36927d41-75a7-4d07-b192-bdaf5294c32f/volumes" Sep 16 01:48:14.173993 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.173959 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6859d7476b-wzdth"] Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174343 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6390385-8fb5-4c70-b967-fc444957e86f" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174355 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6390385-8fb5-4c70-b967-fc444957e86f" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174369 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c32416f-e4bd-4be7-9378-5561418569c9" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174374 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c32416f-e4bd-4be7-9378-5561418569c9" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174386 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b29099ae-0d49-4ec1-9441-2aec696cc971" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174391 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b29099ae-0d49-4ec1-9441-2aec696cc971" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174397 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ddb2492-0829-4518-8fac-6205a79a0d36" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174402 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ddb2492-0829-4518-8fac-6205a79a0d36" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174409 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e48ac20-22c8-44c1-903c-905a54d97b0e" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174414 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e48ac20-22c8-44c1-903c-905a54d97b0e" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174423 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="182f2bdf-91f8-458f-9c7b-a127c0f7cba1" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174427 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="182f2bdf-91f8-458f-9c7b-a127c0f7cba1" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174437 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36927d41-75a7-4d07-b192-bdaf5294c32f" containerName="manager" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174443 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="36927d41-75a7-4d07-b192-bdaf5294c32f" containerName="manager" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174449 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa1c0515-cf74-4231-9141-8028cc2a601f" containerName="maas-api" Sep 16 01:48:14.174464 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174454 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa1c0515-cf74-4231-9141-8028cc2a601f" containerName="maas-api" Sep 16 01:48:14.174951 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174513 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b29099ae-0d49-4ec1-9441-2aec696cc971" containerName="maas-api" Sep 16 01:48:14.174951 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174521 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c32416f-e4bd-4be7-9378-5561418569c9" containerName="maas-api" Sep 16 01:48:14.174951 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174527 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ddb2492-0829-4518-8fac-6205a79a0d36" containerName="maas-api" Sep 16 01:48:14.174951 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174531 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e48ac20-22c8-44c1-903c-905a54d97b0e" containerName="maas-api" Sep 16 01:48:14.174951 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174536 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6390385-8fb5-4c70-b967-fc444957e86f" containerName="maas-api" Sep 16 01:48:14.174951 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174542 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="182f2bdf-91f8-458f-9c7b-a127c0f7cba1" containerName="maas-api" Sep 16 01:48:14.174951 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174547 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa1c0515-cf74-4231-9141-8028cc2a601f" containerName="maas-api" Sep 16 01:48:14.174951 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.174554 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="36927d41-75a7-4d07-b192-bdaf5294c32f" containerName="manager" Sep 16 01:48:14.177444 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.177428 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:14.180792 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.180762 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 01:48:14.180916 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.180793 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 01:48:14.180916 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.180793 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kw9lw\"" Sep 16 01:48:14.180916 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.180762 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 01:48:14.187388 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.187364 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6859d7476b-wzdth"] Sep 16 01:48:14.352994 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.352959 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-webhook-cert\") pod \"maas-controller-6859d7476b-wzdth\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:14.353174 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.353057 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vml6l\" (UniqueName: \"kubernetes.io/projected/aff03787-3a12-490e-910e-3139a52a8544-kube-api-access-vml6l\") pod \"maas-controller-6859d7476b-wzdth\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:14.353174 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.353073 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-metrics-certs\") pod \"maas-controller-6859d7476b-wzdth\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:14.453846 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.453747 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-webhook-cert\") pod \"maas-controller-6859d7476b-wzdth\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:14.453846 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.453818 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vml6l\" (UniqueName: \"kubernetes.io/projected/aff03787-3a12-490e-910e-3139a52a8544-kube-api-access-vml6l\") pod \"maas-controller-6859d7476b-wzdth\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:14.453846 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.453837 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-metrics-certs\") pod \"maas-controller-6859d7476b-wzdth\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:14.456624 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.456592 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-metrics-certs\") pod \"maas-controller-6859d7476b-wzdth\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:14.456727 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.456592 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-webhook-cert\") pod \"maas-controller-6859d7476b-wzdth\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:14.461401 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.461381 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vml6l\" (UniqueName: \"kubernetes.io/projected/aff03787-3a12-490e-910e-3139a52a8544-kube-api-access-vml6l\") pod \"maas-controller-6859d7476b-wzdth\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:14.488980 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.488939 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:14.764323 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:14.764295 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6859d7476b-wzdth"] Sep 16 01:48:14.766636 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:48:14.766595 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff03787_3a12_490e_910e_3139a52a8544.slice/crio-535ffb4302c4840ed7639bba14cfd04a2f047dc000faf91d00218c798bdeccb7 WatchSource:0}: Error finding container 535ffb4302c4840ed7639bba14cfd04a2f047dc000faf91d00218c798bdeccb7: Status 404 returned error can't find the container with id 535ffb4302c4840ed7639bba14cfd04a2f047dc000faf91d00218c798bdeccb7 Sep 16 01:48:15.596632 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:15.596579 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-wzdth" event={"ID":"aff03787-3a12-490e-910e-3139a52a8544","Type":"ContainerStarted","Data":"f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63"} Sep 16 01:48:15.597009 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:15.596681 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-wzdth" event={"ID":"aff03787-3a12-490e-910e-3139a52a8544","Type":"ContainerStarted","Data":"535ffb4302c4840ed7639bba14cfd04a2f047dc000faf91d00218c798bdeccb7"} Sep 16 01:48:15.597009 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:15.596712 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:15.614092 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:15.614042 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6859d7476b-wzdth" podStartSLOduration=1.219813031 podStartE2EDuration="1.614027567s" totalImagesPullingTime="394.214536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:48:15.611340063 +0000 UTC m=+1946.197142274" watchObservedRunningTime="2026-09-16 01:48:15.614027567 +0000 UTC m=+1946.199829778" Sep 16 01:48:26.604434 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:26.604398 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:38.157691 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.157654 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6859d7476b-wzdth"] Sep 16 01:48:38.158184 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.157870 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6859d7476b-wzdth" podUID="aff03787-3a12-490e-910e-3139a52a8544" containerName="manager" containerID="cri-o://f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63" gracePeriod=30 Sep 16 01:48:38.405473 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.405444 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:38.471670 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.471590 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vml6l\" (UniqueName: \"kubernetes.io/projected/aff03787-3a12-490e-910e-3139a52a8544-kube-api-access-vml6l\") pod \"aff03787-3a12-490e-910e-3139a52a8544\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " Sep 16 01:48:38.471813 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.471706 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-metrics-certs\") pod \"aff03787-3a12-490e-910e-3139a52a8544\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " Sep 16 01:48:38.471813 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.471754 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-webhook-cert\") pod \"aff03787-3a12-490e-910e-3139a52a8544\" (UID: \"aff03787-3a12-490e-910e-3139a52a8544\") " Sep 16 01:48:38.473941 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.473900 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aff03787-3a12-490e-910e-3139a52a8544" (UID: "aff03787-3a12-490e-910e-3139a52a8544"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:48:38.473941 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.473902 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff03787-3a12-490e-910e-3139a52a8544-kube-api-access-vml6l" (OuterVolumeSpecName: "kube-api-access-vml6l") pod "aff03787-3a12-490e-910e-3139a52a8544" (UID: "aff03787-3a12-490e-910e-3139a52a8544"). InnerVolumeSpecName "kube-api-access-vml6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 01:48:38.474147 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.473957 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "aff03787-3a12-490e-910e-3139a52a8544" (UID: "aff03787-3a12-490e-910e-3139a52a8544"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 01:48:38.573318 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.573258 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vml6l\" (UniqueName: \"kubernetes.io/projected/aff03787-3a12-490e-910e-3139a52a8544-kube-api-access-vml6l\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:48:38.573318 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.573315 2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-metrics-certs\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:48:38.573318 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.573324 2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aff03787-3a12-490e-910e-3139a52a8544-webhook-cert\") on node \"ip-10-0-142-56.ec2.internal\" DevicePath \"\"" Sep 16 01:48:38.707355 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.707314 2567 generic.go:356] "Generic (PLEG): container finished" podID="aff03787-3a12-490e-910e-3139a52a8544" containerID="f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63" exitCode=0 Sep 16 01:48:38.707523 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.707365 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-wzdth" Sep 16 01:48:38.707523 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.707391 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-wzdth" event={"ID":"aff03787-3a12-490e-910e-3139a52a8544","Type":"ContainerDied","Data":"f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63"} Sep 16 01:48:38.707523 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.707442 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-wzdth" event={"ID":"aff03787-3a12-490e-910e-3139a52a8544","Type":"ContainerDied","Data":"535ffb4302c4840ed7639bba14cfd04a2f047dc000faf91d00218c798bdeccb7"} Sep 16 01:48:38.707523 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.707471 2567 scope.go:117] "RemoveContainer" containerID="f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63" Sep 16 01:48:38.718099 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.718068 2567 scope.go:117] "RemoveContainer" containerID="f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63" Sep 16 01:48:38.718537 ip-10-0-142-56 kubenswrapper[2567]: E0916 01:48:38.718499 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63\": container with ID starting with f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63 not found: ID does not exist" containerID="f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63" Sep 16 01:48:38.718639 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.718549 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63"} err="failed to get container status \"f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63\": rpc error: code = NotFound desc = could not find container \"f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63\": container with ID starting with f87bd1cd60b31e224bfafa0af490dbd49ae93bf55804222edc991723a1e22a63 not found: ID does not exist" Sep 16 01:48:38.733865 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.733833 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6859d7476b-wzdth"] Sep 16 01:48:38.736631 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:38.736605 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6859d7476b-wzdth"] Sep 16 01:48:39.584852 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.584815 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6859d7476b-wqgd5"] Sep 16 01:48:39.585221 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.585200 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff03787-3a12-490e-910e-3139a52a8544" containerName="manager" Sep 16 01:48:39.585221 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.585211 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff03787-3a12-490e-910e-3139a52a8544" containerName="manager" Sep 16 01:48:39.585321 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.585301 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff03787-3a12-490e-910e-3139a52a8544" containerName="manager" Sep 16 01:48:39.589486 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.589467 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:39.591902 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.591880 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 01:48:39.592027 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.591881 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 01:48:39.593006 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.592986 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kw9lw\"" Sep 16 01:48:39.593061 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.592992 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 01:48:39.599830 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.599810 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6859d7476b-wqgd5"] Sep 16 01:48:39.684043 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.684008 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95bd20d4-3a0a-46f1-b4b6-601eee9363b5-webhook-cert\") pod \"maas-controller-6859d7476b-wqgd5\" (UID: \"95bd20d4-3a0a-46f1-b4b6-601eee9363b5\") " pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:39.684188 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.684080 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95bd20d4-3a0a-46f1-b4b6-601eee9363b5-metrics-certs\") pod \"maas-controller-6859d7476b-wqgd5\" (UID: \"95bd20d4-3a0a-46f1-b4b6-601eee9363b5\") " pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:39.684188 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.684163 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9kw\" (UniqueName: \"kubernetes.io/projected/95bd20d4-3a0a-46f1-b4b6-601eee9363b5-kube-api-access-dm9kw\") pod \"maas-controller-6859d7476b-wqgd5\" (UID: \"95bd20d4-3a0a-46f1-b4b6-601eee9363b5\") " pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:39.785488 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.785447 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95bd20d4-3a0a-46f1-b4b6-601eee9363b5-webhook-cert\") pod \"maas-controller-6859d7476b-wqgd5\" (UID: \"95bd20d4-3a0a-46f1-b4b6-601eee9363b5\") " pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:39.785698 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.785507 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95bd20d4-3a0a-46f1-b4b6-601eee9363b5-metrics-certs\") pod \"maas-controller-6859d7476b-wqgd5\" (UID: \"95bd20d4-3a0a-46f1-b4b6-601eee9363b5\") " pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:39.785698 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.785572 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9kw\" (UniqueName: \"kubernetes.io/projected/95bd20d4-3a0a-46f1-b4b6-601eee9363b5-kube-api-access-dm9kw\") pod \"maas-controller-6859d7476b-wqgd5\" (UID: \"95bd20d4-3a0a-46f1-b4b6-601eee9363b5\") " pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:39.788112 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.788084 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95bd20d4-3a0a-46f1-b4b6-601eee9363b5-webhook-cert\") pod \"maas-controller-6859d7476b-wqgd5\" (UID: \"95bd20d4-3a0a-46f1-b4b6-601eee9363b5\") " pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:39.788217 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.788178 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95bd20d4-3a0a-46f1-b4b6-601eee9363b5-metrics-certs\") pod \"maas-controller-6859d7476b-wqgd5\" (UID: \"95bd20d4-3a0a-46f1-b4b6-601eee9363b5\") " pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:39.792996 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.792972 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9kw\" (UniqueName: \"kubernetes.io/projected/95bd20d4-3a0a-46f1-b4b6-601eee9363b5-kube-api-access-dm9kw\") pod \"maas-controller-6859d7476b-wqgd5\" (UID: \"95bd20d4-3a0a-46f1-b4b6-601eee9363b5\") " pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:39.901122 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.901035 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:39.940148 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:39.939804 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aff03787-3a12-490e-910e-3139a52a8544" path="/var/lib/kubelet/pods/aff03787-3a12-490e-910e-3139a52a8544/volumes" Sep 16 01:48:40.173098 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:40.173074 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6859d7476b-wqgd5"] Sep 16 01:48:40.174904 ip-10-0-142-56 kubenswrapper[2567]: W0916 01:48:40.174876 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95bd20d4_3a0a_46f1_b4b6_601eee9363b5.slice/crio-2e1abad8531a443e75354f22147c40294c08704954d16607b01f2250aac94b85 WatchSource:0}: Error finding container 2e1abad8531a443e75354f22147c40294c08704954d16607b01f2250aac94b85: Status 404 returned error can't find the container with id 2e1abad8531a443e75354f22147c40294c08704954d16607b01f2250aac94b85 Sep 16 01:48:40.719038 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:40.718981 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-wqgd5" event={"ID":"95bd20d4-3a0a-46f1-b4b6-601eee9363b5","Type":"ContainerStarted","Data":"2e1abad8531a443e75354f22147c40294c08704954d16607b01f2250aac94b85"} Sep 16 01:48:41.725031 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:41.724991 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6859d7476b-wqgd5" event={"ID":"95bd20d4-3a0a-46f1-b4b6-601eee9363b5","Type":"ContainerStarted","Data":"110683c2a6ea41add8963a34e63cfd7a4c7c1849afee60b5c81edc1935d1b72c"} Sep 16 01:48:41.725509 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:41.725082 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:48:41.741293 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:41.741235 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6859d7476b-wqgd5" podStartSLOduration=2.239944637 podStartE2EDuration="2.741217539s" totalImagesPullingTime="501.272902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 01:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:48:41.739972774 +0000 UTC m=+1972.325774985" watchObservedRunningTime="2026-09-16 01:48:41.741217539 +0000 UTC m=+1972.327019750" Sep 16 01:48:52.732244 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:48:52.732207 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6859d7476b-wqgd5" Sep 16 01:55:43.203866 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:43.203815 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-btbxj_ea88e142-7158-477d-b67a-1f1f450cbc6a/global-pull-secret-syncer/0.log" Sep 16 01:55:43.329983 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:43.329951 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qcv88_5c81a486-0ae1-4caa-be44-9a6cc0edf50c/konnectivity-agent/0.log" Sep 16 01:55:43.412559 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:43.412525 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-56.ec2.internal_af880a3cf392188e514f562daba8d34c/haproxy/0.log" Sep 16 01:55:47.793872 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:47.793839 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bf58fdd4b-qqrcm_f329c94a-0925-4520-842f-0752ab16ffcf/authorino/0.log" Sep 16 01:55:47.891450 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:47.891422 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-vwdb9_bef538a8-b86c-4f3f-872f-a64cbf5391a9/kuadrant-console-plugin/0.log" Sep 16 01:55:49.691099 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:49.691072 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/alertmanager/0.log" Sep 16 01:55:49.711410 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:49.711385 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/config-reloader/0.log" Sep 16 01:55:49.729913 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:49.729883 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/kube-rbac-proxy-web/0.log" Sep 16 01:55:49.749122 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:49.749099 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/kube-rbac-proxy/0.log" Sep 16 01:55:49.766882 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:49.766861 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/kube-rbac-proxy-metric/0.log" Sep 16 01:55:49.785256 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:49.785233 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/prom-label-proxy/0.log" Sep 16 01:55:49.809708 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:49.809689 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c509b351-15b1-4336-866c-12094e7ef184/init-config-reloader/0.log" Sep 16 01:55:49.844513 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:49.844491 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-zs9pd_9ae8823d-2dd3-4b66-a92a-aaca58fad1c6/cluster-monitoring-operator/0.log" Sep 16 01:55:49.985984 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:49.985955 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2pw7_cccf8013-4766-4267-87e7-735069220a4e/node-exporter/0.log" Sep 16 01:55:50.003083 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.003057 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2pw7_cccf8013-4766-4267-87e7-735069220a4e/kube-rbac-proxy/0.log" Sep 16 01:55:50.020841 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.020821 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r2pw7_cccf8013-4766-4267-87e7-735069220a4e/init-textfile/0.log" Sep 16 01:55:50.196432 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.196398 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zk5tk_84bc2eeb-73e5-4859-ae77-6a8543135f2c/kube-rbac-proxy-main/0.log" Sep 16 01:55:50.214735 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.214705 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zk5tk_84bc2eeb-73e5-4859-ae77-6a8543135f2c/kube-rbac-proxy-self/0.log" Sep 16 01:55:50.232360 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.232330 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-zk5tk_84bc2eeb-73e5-4859-ae77-6a8543135f2c/openshift-state-metrics/0.log" Sep 16 01:55:50.267311 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.267223 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/prometheus/0.log" Sep 16 01:55:50.284984 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.284956 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/config-reloader/0.log" Sep 16 01:55:50.305396 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.305372 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/thanos-sidecar/0.log" Sep 16 01:55:50.323793 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.323775 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/kube-rbac-proxy-web/0.log" Sep 16 01:55:50.344407 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.344390 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/kube-rbac-proxy/0.log" Sep 16 01:55:50.363269 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.363247 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/kube-rbac-proxy-thanos/0.log" Sep 16 01:55:50.383509 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.383489 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_950bc2d7-ca5b-476a-875f-f055436c65fd/init-config-reloader/0.log" Sep 16 01:55:50.416755 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.416728 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-pjvgl_4aab2a54-aac1-4a3a-970c-2fd77e17bdeb/prometheus-operator/0.log" Sep 16 01:55:50.435744 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.435722 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-pjvgl_4aab2a54-aac1-4a3a-970c-2fd77e17bdeb/kube-rbac-proxy/0.log" Sep 16 01:55:50.487769 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.487747 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-577b44b7b8-89hvd_7edf63ed-3a7a-4ddc-adef-27f4cd055734/telemeter-client/0.log" Sep 16 01:55:50.507347 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.507326 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-577b44b7b8-89hvd_7edf63ed-3a7a-4ddc-adef-27f4cd055734/reload/0.log" Sep 16 01:55:50.525082 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:50.525022 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-577b44b7b8-89hvd_7edf63ed-3a7a-4ddc-adef-27f4cd055734/kube-rbac-proxy/0.log" Sep 16 01:55:53.446317 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:53.446268 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-gkl42_c28672a2-2930-4f95-80b3-0cccb3ce00d9/download-server/0.log" Sep 16 01:55:54.020224 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.020185 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf"] Sep 16 01:55:54.023663 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.023643 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.026036 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.026012 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hcspv\"/\"openshift-service-ca.crt\"" Sep 16 01:55:54.026159 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.026092 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hcspv\"/\"kube-root-ca.crt\"" Sep 16 01:55:54.027170 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.027149 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hcspv\"/\"default-dockercfg-dfgwp\"" Sep 16 01:55:54.032657 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.032631 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf"] Sep 16 01:55:54.120501 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.120464 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-podres\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.120501 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.120511 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbcq\" (UniqueName: \"kubernetes.io/projected/e09dd548-6391-4ca5-a373-38791f3f9c45-kube-api-access-vvbcq\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.120717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.120584 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-sys\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.120717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.120599 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-lib-modules\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.120717 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.120653 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-proc\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.221476 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.221428 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-sys\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.221476 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.221477 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-lib-modules\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.221751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.221500 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-proc\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.221751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.221547 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-podres\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.221751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.221575 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-sys\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.221751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.221585 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbcq\" (UniqueName: \"kubernetes.io/projected/e09dd548-6391-4ca5-a373-38791f3f9c45-kube-api-access-vvbcq\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.221751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.221626 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-lib-modules\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.221751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.221660 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-proc\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.221751 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.221704 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e09dd548-6391-4ca5-a373-38791f3f9c45-podres\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.229507 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.229482 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbcq\" (UniqueName: \"kubernetes.io/projected/e09dd548-6391-4ca5-a373-38791f3f9c45-kube-api-access-vvbcq\") pod \"perf-node-gather-daemonset-kthsf\" (UID: \"e09dd548-6391-4ca5-a373-38791f3f9c45\") " pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.334183 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.334094 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.602771 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.602723 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf"] Sep 16 01:55:54.767333 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.767294 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" event={"ID":"e09dd548-6391-4ca5-a373-38791f3f9c45","Type":"ContainerStarted","Data":"053b8a997381002af928c615b646ceb808ba8befce833f4fdea23a0c16194323"} Sep 16 01:55:54.767596 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.767350 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" event={"ID":"e09dd548-6391-4ca5-a373-38791f3f9c45","Type":"ContainerStarted","Data":"d18ffa73d1efb541e48bd4f41d97ec3541153a2fb1b4aaaac69ff07cd5f40851"} Sep 16 01:55:54.767596 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.767395 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:55:54.783139 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:54.783090 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" podStartSLOduration=0.783074726 podStartE2EDuration="783.074726ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 01:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 01:55:54.780742543 +0000 UTC m=+2405.366544754" watchObservedRunningTime="2026-09-16 01:55:54.783074726 +0000 UTC m=+2405.368876950" Sep 16 01:55:55.243849 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:55.243820 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6zdx9_2f80a113-872c-49ae-b785-75a115bef17a/dns/0.log" Sep 16 01:55:55.261386 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:55.261359 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6zdx9_2f80a113-872c-49ae-b785-75a115bef17a/kube-rbac-proxy/0.log" Sep 16 01:55:55.368337 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:55.368308 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9glj9_b01ecda8-8841-4584-a12b-daff7da47a47/dns-node-resolver/0.log" Sep 16 01:55:55.983841 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:55.983805 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sxdzj_dc623d5a-e0b2-4aa5-967f-c8b417121518/node-ca/0.log" Sep 16 01:55:57.231809 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:57.231782 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-569b779d49-cpwrh_c4323475-b3bb-457f-84fc-46b6ba632999/kube-auth-proxy/0.log" Sep 16 01:55:58.164724 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:58.164687 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vsrx7_b9a324b1-00bf-4a0d-9e52-bef0adfbcd32/serve-healthcheck-canary/0.log" Sep 16 01:55:58.718977 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:58.718946 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmhgw_799379d2-81b4-458b-9a29-5c43f37c50f1/kube-rbac-proxy/0.log" Sep 16 01:55:58.736329 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:58.736302 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmhgw_799379d2-81b4-458b-9a29-5c43f37c50f1/exporter/0.log" Sep 16 01:55:58.756661 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:55:58.756633 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmhgw_799379d2-81b4-458b-9a29-5c43f37c50f1/extractor/0.log" Sep 16 01:56:00.782336 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:00.782311 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hcspv/perf-node-gather-daemonset-kthsf" Sep 16 01:56:01.309131 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:01.309097 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6859d7476b-wqgd5_95bd20d4-3a0a-46f1-b4b6-601eee9363b5/manager/0.log" Sep 16 01:56:01.491904 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:01.491872 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f84c66984-4889n_fe3f9fc5-8813-403c-9ab6-beb36d75145d/manager/0.log" Sep 16 01:56:01.509708 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:01.509682 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f84c66984-4889n_fe3f9fc5-8813-403c-9ab6-beb36d75145d/copy-manifests/0.log" Sep 16 01:56:02.962551 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:02.962528 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65dff4dd97-jb2bd_d765e931-03cd-4401-866d-497c73323a41/manager/0.log" Sep 16 01:56:02.986193 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:02.986161 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-j7h6t_e3ddae19-090c-4f9d-b920-afc298c8797c/openshift-lws-operator/0.log" Sep 16 01:56:04.402112 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:04.402035 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-569b779d49-cpwrh_c4323475-b3bb-457f-84fc-46b6ba632999/kube-auth-proxy/0.log" Sep 16 01:56:06.359353 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:06.359320 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-569b779d49-cpwrh_c4323475-b3bb-457f-84fc-46b6ba632999/kube-auth-proxy/0.log" Sep 16 01:56:09.343055 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:09.343024 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-78x9p_3ece76da-8177-4048-97cc-df65309482a6/kube-multus-additional-cni-plugins/0.log" Sep 16 01:56:09.361465 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:09.361435 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-78x9p_3ece76da-8177-4048-97cc-df65309482a6/egress-router-binary-copy/0.log" Sep 16 01:56:09.378428 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:09.378397 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-78x9p_3ece76da-8177-4048-97cc-df65309482a6/cni-plugins/0.log" Sep 16 01:56:09.399667 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:09.399639 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-78x9p_3ece76da-8177-4048-97cc-df65309482a6/bond-cni-plugin/0.log" Sep 16 01:56:09.419780 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:09.419753 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-78x9p_3ece76da-8177-4048-97cc-df65309482a6/routeoverride-cni/0.log" Sep 16 01:56:09.436848 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:09.436826 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-78x9p_3ece76da-8177-4048-97cc-df65309482a6/whereabouts-cni-bincopy/0.log" Sep 16 01:56:09.454253 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:09.454228 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-78x9p_3ece76da-8177-4048-97cc-df65309482a6/whereabouts-cni/0.log" Sep 16 01:56:09.795172 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:09.795141 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fxrml_711caeef-4722-4a09-b5df-55262773ebb3/kube-multus/0.log" Sep 16 01:56:09.979955 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:09.979926 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-snt2b_00728728-c5dc-4f6c-b96b-8356103fafa5/network-metrics-daemon/0.log" Sep 16 01:56:09.995096 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:09.995069 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-snt2b_00728728-c5dc-4f6c-b96b-8356103fafa5/kube-rbac-proxy/0.log" Sep 16 01:56:10.802647 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:10.802580 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79tjp_4a1f2830-2236-4dab-bfed-a19ac01c8948/ovn-controller/0.log" Sep 16 01:56:10.816328 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:10.816305 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79tjp_4a1f2830-2236-4dab-bfed-a19ac01c8948/ovn-acl-logging/0.log" Sep 16 01:56:10.836067 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:10.836036 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79tjp_4a1f2830-2236-4dab-bfed-a19ac01c8948/kube-rbac-proxy-node/0.log" Sep 16 01:56:10.854407 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:10.854385 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79tjp_4a1f2830-2236-4dab-bfed-a19ac01c8948/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 01:56:10.870565 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:10.870542 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79tjp_4a1f2830-2236-4dab-bfed-a19ac01c8948/northd/0.log" Sep 16 01:56:10.888815 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:10.888793 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79tjp_4a1f2830-2236-4dab-bfed-a19ac01c8948/nbdb/0.log" Sep 16 01:56:10.908170 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:10.908146 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79tjp_4a1f2830-2236-4dab-bfed-a19ac01c8948/sbdb/0.log" Sep 16 01:56:10.960856 ip-10-0-142-56 kubenswrapper[2567]: I0916 01:56:10.960819 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79tjp_4a1f2830-2236-4dab-bfed-a19ac01c8948/ovnkube-controller/0.log"